0110 — Stamp a grounded "why did Clara say this" rationale on every Clara reply
- Status: Accepted (Gera, 2026-07-22 — "this is good, start driving the PRs")
- Date: 2026-07-22
- Deciders: Gera (owner), Fede
A note on ADR numbers. 19 numbers in
docs/adr/are currently used by two different documents (0079, 0094, 0095, 0101 among them), so a bare "ADR-00XX" can be ambiguous. Cross-references in this document link the exact file.0110was verified free against a same-operation fetch of main, and re-verified after each base update (one011xfile in the merged tree). Both0108(renewal-pipeline) and0109(voice-triage front-door) were claimed on main while this ADR was in review — the collision class this note warns about, twice.
Context
On 2026-07-20, reviewing Clara's leasing conversations, Sean asked why Clara offered a tour on Tuesday instead of the next available Monday. Nobody could answer. Fede's summary was "sometimes it's unexplainable what the LLM does", and the thread closed with a proposal:
Gera: One thing we could add, but it's a little overkill, is a field to every message context why it said what it said. Fede: If it doesn't add latency, that's fine. Gera: It doesn't, because it would be done, like, async. Fede: Yeah, I think we should. […] I think that will be very useful, actually, for me. Because sometimes I'm like, why the […] did you decide this? And it's impossible to know.
The value is triage speed. Gera named the three outcomes the field discriminates, and they map to three different fixes:
| What the rationale reveals | The actual fix |
|---|---|
| The prompt told her to do this | Fix the prompt |
| Two inputs contradicted, she picked one | Fix the data / the precedence rule |
| She made it up | Upgrade the model |
Today, distinguishing those three costs an engineer a manual dig, and only an admin can do it.
What exists and why it isn't enough. AgentTrace (src/lib/data/types.ts:4549) already captures
a great deal per turn: the exact system prompt on the first claude_call
(agents/clara/lib/agent/conversation-manager.ts:4178, default-ON in prod), every tool call and
result, assistantText, token counts, and stopReason. Its own code comment states the motivation
we are extending: "'Why did Clara do that?' is unanswerable without the prompt that produced the
decision." Three gaps remain:
- Inputs, not self-explanation. Everything stored is an input (prompt, tool inputs, catalog
size) or a raw output artifact.
assistantTextis incidental prose the model happened to emit next to atool_useblock — not a solicited, structured account of why this reply was chosen. A reader reconstructs the "why" by reading a 5–15KB prompt plus a step list. - No
Message↔ trace link.Messagehas notraceId.Conversation.agentTraceId(src/lib/data/types.ts:4064) carries a stale doc comment claiming it was deleted in ADR-0006, but it does have a live writer — the voice path sets it atsrc/lib/integrations/voice/trace-persistence.ts:31. It is conversation-level and voice-only, so it cannot answer "which trace produced this message." For text turns the only correlation isAgentTrace.conversationIdplus positional timestamp matching (src/app/api/conversations/[id]/trace/route.ts:52-63). - Wrong audience and granularity. Traces are admin-only, turn-scoped, and fire-and-forget
(
conversation-manager.ts:6020) — a trace can simply be missing. The people asking "why did she say that?" are reading the conversation thread, not the trace viewer.
The trap this ADR exists to avoid. The literal version of the proposal — after the fact, ask a model "here is the response, why did you say this?" — produces a post-hoc rationalization. The explaining model did not see the system prompt, the tool results, or the retrieved facts, so it will invent a fluent, plausible, and unfalsifiable story. That is strictly worse than no field: it launders a hallucination into an audit surface and would send us to fix the wrong one of the three causes above. Grounding the pass in the turn's real trace is therefore not a refinement — it is the whole feature.
Decision
Add an optional, best-effort rationale to Clara-authored outbound message rows, generated by an
asynchronous, trace-grounded LLM pass on the existing turn-settled event stream, and surfaced as one
row in the message's existing click-to-expand details panel.
No new entity is introduced; this ADR adds fields to the canonical Message
(src/lib/data/types.ts:3699), so the ADR-0027 entity-classification table is omitted.
D1 — Five fields on Message
/** ADR-0110 — the id of the AgentTrace whose turn produced this row … */
traceId?: string;
/** ADR-0110 — a short, trace-grounded account of WHY Clara sent this reply … */
rationale?: string | null;
/** ADR-0110 — the model that produced `rationale` … */
rationaleModel?: string;
/** ADR-0110 — when `rationale` was generated (ISO) … */
rationaleAt?: string;
/** ADR-0110 D5 — on a refusal, the short clause naming the evidence gap …
* (added by the 2026-07-23 output-polish PR) */
rationaleRefusalReason?: string | null;
rationaleis hard-capped at 280 characters at write time (tightened from the original 600 on Gera's verbosity feedback, 2026-07-23 output-polish PR: the prompt targets one/two sentences under 240 chars; the 280 cap is the backstop, applied gracefully — last full sentence that fits, else last word boundary + ellipsis, never mid-word). It is prose for a human triaging a conversation, not a reasoning dump.nullmeans "the pass ran and declined to explain" (see D5). Absent means it has not run. The two are different and the UI renders both as an honest empty — never a filler string (CLAUDE.md "no fabricated fallback" rule;no-pf-stub-drift.test.ts). Storage-encoding correction (PR-4 review, 2026-07-23): the three states are semantic, but the STORED encoding of "refused" is NOT a persistednull—updateItemFieldsmaps a JSnullto a DynamoDBREMOVE(nulls are illegal on GSI key attributes), so a refused row reads back withrationaleABSENT. The durable "visited" sentinel isrationaleAt(a real ISO string on every stamped-or-refused row): visited =rationaleAtpresent; within visited, stamped =rationaleis a non-empty string, refused = it is not. Every consumer (eligibility, the D6 conditional, the coverage detector, the UI three-state) keys onrationaleAt, never on'rationale' in row. Since the output-polish PR, a refusal may additionally carryrationaleRefusalReason(≤200 chars) — the model's short clause naming the evidence gap (CANNOT_EXPLAIN: <clause>), rendered in the "Why" row as "Couldn't ground this reply —" in the placeholder register. Rows refused before that PR have no reason and keep the em-dash; rationaleitself remains strictly grounded-explanation-or-null.rationaleModelexists because the field's job is to tell us whether to change models. Fede is already evaluating Sonnet 5; a rationale corpus you cannot attribute to a model cannot answer "did the upgrade help?" It is ~20 bytes.rationaleAtexists for the same reason applied to prompts: after a prompt fix ships, the first triage question is "was this bad rationale generated before or after the fix?" — andpreserveUpdatedAt: true(D6) deliberately keeps the row'supdatedAtfrom recording the stamp time, so without this field the question is unanswerable. ~24 bytes.- Deliberately not stored: token counts or the reasoning chain. The field is cheap, disposable, and regenerable; the generating activity logs the rest. This is the direct answer to Gera's own "it does add a lot of bloat" caveat.
- Field naming avoids
sourceanddirection, which are banned asMessagemembers by F3 insrc/__tests__/conversation-single-writer.drift.test.ts:181.
D2 — Scope: keyed on traceId, not on authorship
The pass stamps a row only when all hold:
- the row carries a
traceId(stamped per D3), and - the resolved trace contains at least one real
claude_call, and - it is an outbound row, not an audit row and not
turnScope: 'pm_query', and rationaleis absent.
Why author.kind === 'clara' is NOT the scope key (it was, in an earlier draft of this ADR —
two code facts broke it):
- Clara-authored ≠ LLM-chosen. The agent loop has at least one path that writes a
Clara-authored outbound with no Claude call at all — the language-ask at
conversation-manager.ts:3670-3690, whose own comment reads "trace id absent because no Claude call was made" and which sets_lastAgentResult.traceId = ''. A rationale on a deterministic template row is confabulation about a non-decision. - Voice rows are Clara-authored too (
api/voice/call-ended/route.ts:693) but their traces are built post-hoc from the ElevenLabs transcript (voice-trace-builder.ts:141-175): syntheticclaude_callsteps and no captured system prompt — the EL agent's prompt lives in ElevenLabs, not in the trace. The grounding evidence D3 requires simply is not there.
Keying on traceId makes the scope self-describing: a row is explainable iff the turn that
produced it left real evidence. It also gives us the epoch bound for free — traceId stamping
begins when PR-1 ships, so historical rows are out of scope by construction, and the first settle
on a 200-message thread cannot trigger a paid explain-the-backlog sweep. (The topics rollout needed
an explicit CLASSIFICATION_EPOCH for exactly this; here the link field is the epoch.)
Voice is deferred, not rejected. Explaining voice replies is a V2 with its own evidence assessment (what the EL payload can actually ground) — not a silent inclusion with thinner evidence and the same confident prose.
Also out of scope: inbound tenant messages (there is no "why" to explain), tool-audit rows, and
system-triggered templated sends — those already carry triggerSource, which the UI renders as
a deterministic "Sent because: A new work order was created" (ConversationThread.tsx:459-465). An
LLM rationale on a row whose cause is already known deterministically is pure cost and pure risk.
The human/machine line, stated as a rule (Gera, 2026-07-22): rationale marks MACHINE choices
only. Every human-side message — tenant, vendor, PM, and the subtle one, operator-typed
outbound (Pipeline Lab's "Send as Clara", Message.kind: 'manual_outbound', which renders on
Clara's side of the thread but is human-authored) — never carries a traceId and therefore can
never be stamped, by construction rather than by filter. A human message is its own explanation;
an AI-generated one owes you one.
D3 — Grounding (the load-bearing decision)
The activity MUST assemble its prompt from the turn's real inputs, not from the reply text alone:
- Resolve the turn's
AgentTracefor the message. - Feed the model: the captured
systemPrompt, the ordered tool calls with their inputs and results,assistantText, and the inbound message — then the reply to be explained. - Instruct it to explain from that evidence, cite which input drove the choice, and return the refusal sentinel (D5) when the evidence does not support an explanation.
Resolving the trace requires a link that does not exist today. Two options, and the ADR picks the first:
- (chosen) Stamp
traceIdon the message row, threaded through the write spec at build time. The trace id is minted at collector construction (trace-collector.ts:85,id: generateId('trace')) — i.e. it exists at turn start, before the reply row is persisted atconversation-manager.ts:3685. So the agent loop passes it on theOutboundSpecthroughbuildMessage, the same waywriterIdtravels. This is deliberately NOT a backstamp from_lastAgentResult(:6027): that global is set only after trace finalize, would race the row write, and is a module-global the stamp should not depend on. Spec-time threading is atomic with the row itself, independently valuable (every message becomes one hop from its trace), and gives the activity an O(1) trace lookup. - (rejected) Positional matching on
conversationId+ timestamp, asapi/conversations/[id]/trace/route.tsdoes for the turn picker. It is inherently fuzzy, and silently mis-attributing a rationale to the wrong turn is exactly the confabulation failure this ADR is trying to prevent.
A real ordering race exists and the design must absorb it. The turn-settled event is emitted at
conversation-manager.ts:5934 — before the trace is finalized (:6000) and before
saveAgentTrace runs inside a fire-and-forget IIFE (:6020). The debounce (D4: 10s as of
2026-08-28, previously 2 min) covers this in the common case, but "usually covers it" plus "no
reconciler" equals permanent gaps that concentrate exactly when the system is under load — which is
exactly why the retryable-vs-refused split below exists rather than leaning on the debounce alone.
So trace-absence is split into two outcomes:
- Row has
traceId, trace row not found → retryable. The activity throwsApplicationFailure{ nonRetryable: false }so Temporal's 3 attempts spread over minutes give the fire-and-forget save time to land. Only after retries exhaust does the row stay unexplained. - Evidence resolved but insufficient → the D5 refusal, written durably as
rationaleAt+rationaleModelwith norationalestring (see the D1 storage-encoding note).
What the pass never does is fall back to reply-text-only explanation. A missing rationale is honest; a confabulated one is not.
D4 — Async transport: the existing turn-settled stream
Reuse ADR-0088 end to end. No new cron, no new sweeper (CLAUDE.md:47, :108; the topics reconciler was deliberately deleted by ADR-0094 delete-conversation-reconcilers).
turn settles → onConversationTurnComplete (turn-complete-dispatch.ts:86)
→ SQS propflow-conversation-events.fifo (IDs/enums only, PII-fenced)
→ lambda/conversation-events-bridge/registry.ts ← ONE new consumer entry
→ signalWithStart conversationRationaleWorkflow (workflowId: conversation-rationale-<conversationId>)
→ debounce, then one activity
→ activity reads DDB + trace, calls Anthropic, patches each MSG# row
This satisfies Fede's only condition — zero added latency on the reply path. The reply is
already sent before the event is emitted, and the emit is a ~20–50ms intra-AWS SQS publish that the
agent loop awaits (conversation-manager.ts:5933-5950).
Three constraints are inherited and non-negotiable:
- In-process fire-and-forget is not an option. The text settle tail runs in the
inbound-processor Lambda, which freezes its container the moment the handler resolves; a dangling
promise may never run (
turn-complete-dispatch.ts:20-22). Next.jsafter()was tried and abandoned (PR #1338). - The Lambda bundle must stay Temporal-free. Signalling happens only from the
conversation-events-bridge Lambda, the one runtime allowed to be a Temporal client on this path
(F6,
conversation-single-writer.drift.test.ts:264). - New signal fns get exactly one caller — the registry (
turn-settled-single-path.drift.test.ts).
Task queue: new propflow-conversation-rationale, co-tenanted in the existing Fargate worker
process via getRationaleWorkerRegistration() — the same "no infra change" pattern as
conversation-topics-worker.ts:5-8.
Debounce: mirror the topics workflow — 10-second idle debounce, 60-second hard cap (cut
from 2 min / 10 min on 2026-08-28, conversation-topic-stamp-workflow.ts — same owner call: don't
hold a customer-visible "Explaining…" row on a wait bought for LLM-call batching savings that
aren't worth it). The shorter window means the trace-save ordering race above (D4 intro) resolves
via retry more often than via debounce absorption — already-designed-for, since the activity's
nonRetryable:false + Temporal's 3 retries spread over minutes exist for exactly this case. Because
the stream coalesces, the activity must work in the applyTopicStampPlan shape: query the
conversation, select every qualifying row still lacking rationale, and explain each. It is not
"one event, one message."
D5 — Failure and honesty policy
- Refusal is a first-class outcome. When the trace does not support an explanation, the model
returns a sentinel and we record a durable "we looked and could not say" (
rationaleAt+rationaleModel, no rationale string — see the D1 storage-encoding note). This is what keeps the field trustworthy; a pass that always produces prose is a pass that confabulates. It is also the defensible posture for a written record: rationales are durable, discoverable artifacts, and a grounded refusal reads far better in any future dispute than a confident invention. - Skip
isTestproperties so eval and sandbox traffic never burn LLM spend (conversation-topic-classification.ts:80-86). - Never overwrite an existing
rationale(mirrorsapplyTopicTags:346-347). - Fail-soft, retried, then dropped. Activity errors raise
ApplicationFailure{ type:'TransientUpstream', nonRetryable:false }; Temporal retries 3×; then the edge is dropped. Per ADR-0094 delete-conversation-reconcilers there is no reconciler backstop, so a dropped edge means that message stays permanently unexplained. The field is therefore honestly nullable by design, and no consumer may assume coverage. - No env arm. Per CLAUDE.md rule 13, this ships working. The controls that already exist —
the
isTestskip, the per-conversation debounce, and a worker deploy — are sufficient; a global arm would just create a silent-dark failure mode. - Coverage is watched, even though it is never repaired by a sweep. A best-effort field with no
observability degrades silently — if PR-1's stamp regresses or traces stop resolving, coverage
drops to zero and nobody notices (the exact failure the
classification_blank_topicdetector exists for,classification-observability.detector.ts:64,106). Register rationale coverage on the same daily detector surface: page when the stamped-rationale rate over trailingtraceId-bearing rows falls below threshold. A read-only detector is not a reconciler; it pages a human instead of silently rewriting data. - Regeneration is an operator-run one-shot, never a cron. "No reconciler" (ADR-0094) bans
standing sweeps, not deliberate one-shot scripts — the repo is full of sanctioned
scripts/backfill-*.ts/scripts/drain-*.tstools with dry-run defaults. After a rationale prompt fix,scripts/backfill-message-rationales.ts(dry-run default, bounded, explicit--mode=apply) is the path to re-explain a window of rows;rationaleAt+rationaleModelare what make "which era produced this rationale" answerable afterwards. - Model + prompt budget are pinned in the pure module. The pass defaults to
AGENT_MODEL(the topics-classifier precedent,message-topic-classifier.ts:370-388), overridable per-call — that is the "cheaper model" cost lever. The prompt builder hard-caps each evidence section (per-tool-result, assistantText, inbound, reply) the wayMAX_TEXT_PREVIEW = 4000bounds trace text (trace-collector.ts:28), so a turn with a 60KB tool result cannot produce an unbounded prompt. Both live in the pure, unit-tested builder. - The system prompt evidence is RELEVANCE-SELECTED sections, never a blind prefix
(revised 2026-07-27 after a prod false-refusal incident). The original budget was a flat
6000-char prefix of the captured system prompt. Real Clara system prompts measure ~80–100KB
(two prod traces: 99,901 and 79,690 chars), so the pass read ~6% of the prompt as an arbitrary
prefix — and twice refused with a confident fabrication accusation against a reply that quoted
the prompt's
ACTIVE LEASING SPECIALSsection verbatim (the section sat at char offset ~10,669, past the cut). A false fabrication finding is the worst possible output for this feature: the refusal channel exists to catch real hallucinations, not to indict correct behavior. The fix (inmessage-rationale.ts): segment the prompt by its real header convention (markdown##headings + column-0 ALL-CAPS runs closed by:/—), score each section's relevance to the specific reply with a pure, deterministic lexical scorer (weighted token overlap — reply terms dominate — plus reply-bigram phrase hits, length-normalized), and select top-scoring sections into a 24,000-char budget, always including the identity preamble and the pinned turn-context sections (CURRENT CALLER/ACTIVE CAPABILITIES/RETURNING TENANT— the constraint-driven rationales are grounded in exactly those). Every omission is announced in place ([… N prompt sections omitted …], the same never-silent instinct as themaxToolStepsdrop marker), the evidence header labels the excerpt as a subset, and the refusal guidance tells the model that "not in the SHOWN sections" must never be voiced as "Clara invented it" — the gap-reason on a subset refusal says the claim may live in an unshown prompt section. Headerless prompts fall back to a budget-capped prefix, still marked truncated. Subset means ANY missing content (revised again post-merge on the #4691 review): a section or preamble truncated in place by its per-piece cap counts exactly like an omitted section — partially shown is not fully shown — and the 24,000-char budget is a hard ceiling with markers and joins charged inside it, not a soft target the assembly overshoots. Selection quality is observable per row via the counts-only[MessageRationale] evidence+outcomelog line (IDs + selection counts + outcome kind, no prompt/reply text per ADR-0026). Known remaining gap (V2, out of scope here): evidence is single-turn, so a reply grounded in a PRIOR turn's tool results still reads as ungrounded.
D6 — Persistence: targeted per-message patch
Messages are separate DDB items (PK=CONV#{convId}, SK=MSG#{timestamp}#{msgId},
conversation.ts:4-13), so this is a keyed UpdateItem, not a conversation rewrite. Follow the
existing applyTopicStampPlan / backstampProviderId precedent
(conversation-writer.ts:478-495, :729-751) exactly:
await updateItemFields(convPK(conversationId), convMsgSK(timestamp, msgId),
{ rationale, rationaleModel, rationaleAt },
{ preserveUpdatedAt: true,
// First-stamp-wins keys on rationaleAt, NOT rationale: a refusal's
// `rationale: null` becomes a DDB REMOVE (never a stored attribute), so
// attribute_not_exists(rationale) would always pass on refused rows and
// let a concurrent retry re-stamp + re-bill the LLM (PR-4 review fix).
conditionExpression: 'attribute_exists(PK) AND attribute_not_exists(rationaleAt)' });
Three rules the implementation must not miss:
- Carry
{ conversationId, msgId, timestamp }in the plan. The SK needs both halves;findMessageByMsgIdrecoverstimestampvia a fullgetConversation()(conversation-writer.ts:470-473), which would make every stamp an O(conversation) read. attribute_exists(PK)is mandatory —UpdateItemupserts by default and would mint a phantomMSG#row (conversation.ts:868-874).preserveUpdatedAt: true—MSG#rows are append-only history; a metadata patch is not an edit (conversation.ts:890-893).
saveConversation is append-only for messages and skips existing msgIds, so re-saving the
conversation cannot be used to apply this.
D7 — PII boundary
The rationale is free text derived from tenant conversation content, so under ADR-0026 it must not appear in workflow inputs, signal payloads, or activity return values — Temporal Cloud workflow history is plaintext to anyone with namespace read access.
- Workflow input: IDs only.
- The rationale is generated inside the activity and written straight to DDB from there — the
shape
stampConversationTopicsActivityalready uses. - The activity's return type is counts and enums only (
{ examined, stamped, refused, skipped }), mirroringStampConversationResult(conversation-classification.ts:280-287). - Any log line carrying it goes through
sanitizeForLog/maskPhone(src/lib/platform/security/sanitize.ts).
Containment: the field is one-way, operator-facing display data. Two invariants keep it on the inside of the product:
- Never outbound. No composer, template, or send path (SMS / email / voice / Telegram) may ever
read
rationaleinto a message body. It describes internal reasoning — prompt policy, input contradictions — and it belongs on the operator dashboard only. The tenant-facing channels never see the field. - Never an LLM input.
rationaleis metadata about a past decision, not conversation content — it must never be replayed into Clara's context (buildClaudeMessagesmaps role/content only, which keeps this true today; keep it that way) and never fed into the rationale pass's own prompt for later turns. No self-reference loop: explanations don't explain explanations.
D8 — UI surface
One row in the existing per-message details panel — no new component, no dialog. buildMessageDetails
(ConversationThread.tsx:361) already renders a two-block "show the logic" panel, and the click
target already exists (the avatar doubles as the details button; ChatSurface.tsx:769-773).
Add a "Why" row to the Provenance block, as a sibling of the deterministic "Sent because":
Provenance
Why Offered Tuesday because the 24h notice window ruled out Monday
Sent by Clara
Channel SMS · Sent at 3:41 PM
- The "Why" row is pushed only for rationale-eligible rows (
traceIdpresent). On human messages — tenant inbound, vendor, PM,manual_outbound— the row is omitted entirely, not rendered empty: an em-dash "Why" on a human bubble would imply the system might explain humans.buildMessageDetailsalready pushes rows conditionally ("Sent because" appears only on system-triggered sends), so this is the existing convention, not a new one. - On eligible rows the Why row has four states (
ConversationThread.tsx, keyed onrationaleAtper D6's visited-sentinel rule):rationaleis a non-empty string → the grounded explanation, normal register.rationaleAtset +rationaleRefusalReasonpresent → "Couldn't ground this reply — <reason>" in theplaceholder: truefaint register — context, visually distinct from a real explanation (added by the output-polish PR).rationaleAtset, no reason (the pre-polish refused cohort, or a bare-sentinel refusal) → the honest em-dash placeholder.- No
rationaleAt→ "Explaining…" pending placeholder (the pass hasn't visited the row).
- The rationale is written in English regardless of conversation language. Its audience is the
operator reading the dashboard, not the tenant — the conversation itself may run in Spanish
(ADR-0089
languageprovenance), but the field is operator-side metadata, not conversation content, and one consistent language keeps the corpus greppable. - Pipeline Lab inherits this for free (it calls the same builder,
pipeline-lab/page.tsx:2030). - Per CLAUDE.md, any pending state uses
ClaraThinking, never a generic spinner;CLASSIFYING_LABEL(ConversationThread.tsx:40) is the existing precedent for "the pass hasn't run yet."
Consequences
Easier. Every Clara reply becomes self-describing to whoever is already reading the thread —
Fede's stated need. Triage moves from "dig through a 15KB prompt in the admin trace viewer" to
"expand the message." The three failure classes become separable at a glance, and because
rationaleModel is stamped, a model migration becomes measurable rather than vibes-based.
Stamping traceId on the message row (D3) is independently useful: it makes every message one hop
from its full trace and retires the fuzzy positional correlation.
Harder / accepted costs.
- One extra LLM call per Clara reply, on the highest-volume path in the product. This is the main cost of the feature and it is real. No cost estimate is asserted here — it must be measured before the pass is enabled fleet-wide, by counting qualifying rows over a representative window and pricing the pass at its configured model. Levers if it bites: a cheaper model for the pass (it is a summarization task over supplied evidence, not a reasoning task), batching a turn's rows into one call, or restricting to properties under active review.
- Row growth. ≤280 chars (+ ≤200 for a refusal's gap-reason) on a
MSG#row that is otherwise ~1KB. Safe against the 400KB item ceiling precisely because messages are separate items. If the payload ever grows beyond a short paragraph, move it to a siblingRATIONALE#<msgId>row rather than inflatingMSG#— themaestro-eval-run.ts:254-261pattern. - Coverage is permanently partial. No reconciler is permitted, so dropped edges leave gaps forever. Any future analytics over this field must treat absence as unknown, never as "no reason."
- Confabulation risk is reduced, not eliminated. D3 grounding and D5 refusal are the mitigations; the eval (below) is what keeps them honest. If the eval shows the pass inventing causes not present in the trace, the correct response is to tighten the refusal sentinel, not to ship it anyway.
- Two type trees. The field must be added to
src/lib/data/types.tsandagents/clara/lib/data/types.tswith the lockstep breadcrumb, because the Lambda inbound funnel runs the agent loop.message-types-lockstep.drift.test.tsis a per-field text test and will not auto-catch a new field — a newit()pin is required.scripts/generate-entity-schemas.tsmust be re-run and its output committed.
Follow-up implied. The eval-gate path map does not currently watch
src/lib/domain/conversations/** — editing the classifier there does not fire the promptfoo sweep.
If the rationale pass lands under that path, add it to .github/workflows/clara-behavior-evals.yml
and to CLARA_PATH_MAP in src/__tests__/clara-eval-gate-path-filters.test.ts in the same PR.
Two stale doc comments in src/lib/data/types.ts are worth one-line corrections:
AgentTraceStep.systemPrompt (:4456-4460) still claims prod skips prompt capture, contradicting
the live code, and Conversation.agentTraceId (:4064) claims the field was deleted in ADR-0006
while the voice path writes it live (trace-persistence.ts:31). Finally, voice rationale support
is an explicit V2 (D2): it needs its own evidence assessment of what the ElevenLabs payload can
ground before any voice row gets explained.
Drift guards + evals
| Deliverable | Where |
|---|---|
| Field pin, both trees | new it() in src/__tests__/message-types-lockstep.drift.test.ts |
| Single-caller signal fn | existing turn-settled-single-path.drift.test.ts (new consumer must comply) |
| Writer discipline | existing F2/F6/F7 in conversation-single-writer.drift.test.ts |
| Generated schema refresh | scripts/generate-entity-schemas.ts output committed |
| Pure prompt-build + parse unit tests | src/__tests__/message-rationale.test.ts |
| Promptfoo config, provider pinned to production model/temp/max_tokens | evals/promptfooconfig-message-rationale.yaml |
| Cases — one per failure class, incl. cases where refusal is REQUIRED | evals/datasets/message-rationale.yaml |
| Prompt bridge importing the real builder (never a copy) | evals/prompts/message-rationale.ts |
| Grader (LLM-as-judge; fairhousing + maintenance-judge are the in-repo precedents) | evals/lib/assert-message-rationale.ts |
run_eval registration |
evals/run-all.sh |
| Coverage detector (silent-degradation page) | classification-observability.detector.ts sibling registration |
| Operator regeneration tool (dry-run default) | scripts/backfill-message-rationales.ts |
The eval's central assertion is groundedness: every causal claim in the rationale must be traceable to something in the supplied trace evidence, and an unsupported reply must produce the refusal sentinel rather than prose.
Suggested PR sequence
- PR-1 —
traceIdonMessage(both trees + pin + generated schema), threaded through theOutboundSpecatbuildMessagetime (D3 — not a backstamp). Independently useful; unblocks D3 and defines the scope epoch (D2). - PR-2 —
rationale/rationaleModel/rationaleAtfields + the pure prompt-build/parse module (caps included) + unit tests. No wiring, nothing runs. - PR-3 — eval dataset + grader green. Gates PR-4.
- PR-4 — workflow, activity, task queue, bridge-registry consumer, coverage detector registration.
- PR-5 — the UI "Why" row.
- PR-6 (as needed, post-ship) —
scripts/backfill-message-rationales.ts, the operator-run regeneration tool.
Alternatives considered
Ask the model post-hoc from the reply text alone (the literal proposal in the meeting). Cheapest and simplest, and rejected as the primary design: without the prompt and tool results the explaining model cannot know why the reply was chosen, so it produces fluent, unfalsifiable invention. It would have "explained" the Tuesday-vs-Monday bug with a confident story about availability. D3 keeps the async shape Gera specified but grounds it.
Have Clara emit the rationale inline during her turn. Most faithful — it is the actual deciding model with the actual context. Rejected on Fede's explicit condition: it adds output tokens to every reply on the latency-critical path. It also makes the rationale a hostage of the reply (a formatting slip corrupts a tenant-facing message), and self-reported reasoning from a model choosing its own answer is not notably more truthful than a grounded reconstruction.
Just improve the existing trace viewer. Zero new storage and no new LLM call. Rejected because it does not address the audience or granularity gaps: the people asking the question are reading the conversation thread, the surface is admin-only, and no amount of viewer polish converts a 15KB prompt into a one-line answer. The trace remains the ground truth this feature is derived from.
A new cron/sweeper to backfill rationales. Explicitly rejected — CLAUDE.md:47 and :108 name this as the wrong direction, and ADR-0094 delete-conversation-reconcilers deleted the equivalent topics reconciler. Coverage gaps are accepted instead.
Reuse the SQS propflow-agent-jobs.fifo lane. Rejected: that queue is the single named path for
AppFolio browser-agent jobs, guarded by agent-name-handler-coverage.test.ts. Using it for LLM
compute would abuse a single-purpose queue.