Clara's Guard Layer — the 10,000-ft Investigation

2026-08-25 · Commissioned by Fede off the Camellia missing-application-link incident (PR #6278 thread) · Every file:line verified against origin/main at head 6167989bc8 (2026-08-25); the incident trace read directly from propflow-prod (read-only)

Contents

1. §1 The incident, exactly — provenance
2. §2 Current state — every guard, verified
3. §3 Gap analysis — the 3 actions, the trace-judge bar
4. §4 Golden dataset — exists vs described
5. §5 The #6278 mechanism — refactor / eliminate / evolve
6. §6 Recommendation + per-guard fit
7. §7 Staged migration + proof gates
8. §8 Decisions — both answered
9. Appendix — verification notes & corrections

§1 The incident, exactly — what change introduced the mechanism, and why it decided the way it did

Fede's ask, verbatim: "I want to know exactly what change introduced the guard that re-wrote the message, why it decided to re-write the 'right' response and what other gaps it has."

The precise answer: no guard re-wrote anything. The mechanism that discarded the correct reply is not a guard at all — it is the agent loop's final-text-extraction seam, and it has a three-commit history:

1.1 Why the 2026-05-27 decision was right then and wrong for Daniel's turn shape

PR #1540 fixed a real defect: on the escalation shape it targeted, the pre-tool text and the final text were near-identical duplicates, so "deliver only the final" was safe — the content survived in the final bubble. The fix's implicit assumption is final ⊇ draft: whatever mattered in the draft is restated in the final reply. That assumption held for the duplicate-bubble shape and is false in general. On Daniel's turn the draft held the substance — the application link (https://www.camelliadenver.com/#floor-plans, pulled verbatim from PROPERTY INFO), the 12-month special, studio ~$1,000 / 1BR ~$1,200 ranges — and the final was a content-free wrap-up ("All set — just let me know your preferred day and time…"). Same seam, opposite content distribution — and the loop deleted the substance. The model's own stored [[WHY]] on the final turn claims it "Provided application link, surfaced the matching 12-month special, presented unit ranges": it genuinely believed the draft had shipped.

1.2 The trace, step by step — proof no guard altered the turn

Read directly from propflow-prod (TRACE#trace_de3ff842-a479-4425-9795-6040ff7b10b7, conv a05c3c99-…, 2026-08-25T19:18Z). All ten steps:

So on this turn: every guard that ran passed the text through unmodified. The typed-tool-call strip recorded no step (no markup). The hallucination guard never ran — it only arms behind a renewal-offer or active-tour snapshot, and a first-contact Zillow lead has neither. The policy gate (the honesty layer) and the leasing fee-fabrication guard — the two mechanisms whose job description covers this claim family — are both disarmed by default and recorded nothing. The discard happened before any guard saw anything: iteration 0's text was never a candidate for delivery in the first place.

One more honest nuance: even armed, the policy gate as specced would likely not have caught this turn. Its judge verifies claims in the draft against evidence; Daniel's delivered reply made no false claim — it omitted the answer. Omission-of-the-ask is a completeness failure, a different check (it needs the inbound message as an input, which the gate today does not receive — §3.3). The thing that catches Daniel's shape deterministically is exactly what #6278 shipped.

§2 Current state — every guard on Clara's outbound path, verified against code

There are two stages. Pre-generation guards decide whether Clara replies at all; the post-generation pipeline mutates or replaces the drafted reply before send. All line numbers are agents/clara/lib/agent/conversation-manager.ts on origin/main (2026-08-25, post-#6278) unless noted.

2.1 Pre-generation / in-loop guards

2.2 The post-generation pipeline — one chokepoint, sixteen mechanisms, fixed order

This is the "centralized layer that runs before Clara responds." It already exists: every text/email reply flows through these in exactly this order.

Reading the table against Fede's vision: the "centralized layer that runs before Clara responds" is not missing — it is rows 1–16, one call site, fixed order. What's fragmented is the decision logic: rows 8, 11–14 are hand-written per-domain checks bolted on over four months of incidents, while row 15 is the general framework they were supposed to converge into — built, spec'd (docs/planning/policy-gate-spec.md, PR #5815), measured offline, and never armed. Row 16 is a second finished-but-dark guard. The architecture source of truth (§4.5, "The honesty layer — history, failure, redesign") already tells this exact story and prescribes the convergence; this doc's job is to confirm the code matches it (it does) and put the arming decision on the table.

§3 Gap analysis — Fede's three actions, then his trace-judge bar

3.1 The three actions: rewrite / flag / auto-heal

3.2 Should the rewrite loop be built? The honest trade

The absence is a decision the codebase already made once, implicitly: post-generation guards are deterministic precisely because they backstop a non-deterministic generator. A regenerate loop reintroduces that non-determinism at the safety boundary, plus:

• Latency: one full agent call (~5–15s SMS/email; disqualifying for voice, where the standing founder rule is zero added live-call latency).
• Loop risk: the rewrite can trip the same guard again (or a different one) — needs a hard retry ceiling and a deterministic post-rewrite re-check, which ends in block/hold anyway on the second miss.
• Verification asymmetry: for claim failures ("I've told the team" with no record), repair is strictly better — it makes the sentence true instead of asking the model to say something weaker. For compliance failures, the classifier's suggestedRewrite already is the rewrite. The class where regeneration genuinely wins is omission/completeness (Daniel's class): the draft isn't wrong, it's incomplete, and no tool can "repair" a missing answer. #6278 covers the mechanical-token subset; a completeness verdict + one bounded regenerate is the natural evolution if the residual (dates, unit numbers, policy answers, yes/no commitments dropped at the seam) shows up in measurement.

Position taken here: don't build it as part of arming; add a revise verdict later, scoped to completeness findings only, one retry, deterministic re-check, never on voice — and only once the offline replay shows the residual class actually occurring. That decision is raised for Fede in §8.

3.3 The trace-judge bar: "inspect the whole trace… second pass against ground truth, independently, as a judge"

Mapping Fede's four inputs against what the built (disarmed) gate actually receives (EvaluatePolicyGateInput, policy-gate-types.ts:143–149):

So the built gate is a trace-level judge in shape — it grades the draft against tool evidence and history, not just against token patterns — but it is a claim verifier, not yet a completeness checker, and two of Fede's four inputs (the inbound, the reasoning) aren't plumbed. Both are additive input changes to PolicyGateEvalInput, not redesigns; the replay harness already reconstructs turns from AgentTrace rows that carry the inbound, so the offline measurement can validate the enriched judge before it ever runs live.

§4 The golden dataset — what exists vs what Fede described

Fede: "Us as the humans we'll be annotating conversations to build a golden data set of what good and bad is which then should serve as our regression suite to avoid bad changes going prod."

Sketch of the missing half (deliberately small): (1) a label action on the conversation page — good / bad + one-line why + which turn — writing a PII-safe pointer row (conversationId + turn index + label, never copied text); (2) the labeled set becomes a replay manifest: the existing replay harness re-runs the guard chain + (offline) the judge over exactly those conversations on every candidate build, asserting bad ones trip and good ones don't; (3) the same rows calibrate the trajectory judge (its spec already calls for a ≥200-case gold set). This turns the annotation habit Fede described into the regression gate he described, using two harnesses that already exist. It also answers Q1: resuming post-hoc judging feeds the candidate queue.

§5 The #6278 mechanism — refactor, eliminate, or evolve?

Fede's third ask: what other gaps does the mechanism have, and does it get refactored, eliminated, or evolved?

5.1 What the seam + salvage still miss

• Fact classes beyond URLs and $ amounts. The salvage counts only mechanically extractable tokens, deliberately ("detecting substance in prose is exactly the thing that had just gone wrong"). Dropped dates, times, unit numbers, policy answers, yes/no commitments, names are not salvaged. A draft "Yes — parking is included, and unit 204 is available for a Tuesday 2pm tour" dropped for a wrap-up final ships the wrap-up.
• Voice. Excluded by design (splicing an earlier draft into a spoken final would double-speak). Voice's answer is the settled post-call reconciliation doctrine (H3), which doesn't exist yet either.
• Tool-failure turns. Salvage fails closed when any tool this turn failed — correct (the draft may describe a reality the failure contradicted), but those turns get nothing: no salvage, no regenerate, and unless the failure was fatal-class (Anomaly Review Gate) the possibly-hollow final ships.
• Multi-draft supersession edge: a segment whose money disagrees with a money-quoting final is skipped wholesale — right for the fee-fabrication class, but its non-money facts (a URL in the same paragraph) are lost with it. Accepted trade-off, documented in the PR.
• Visible duplication when the final restates the URL but drops the prices (whole segment merges). Accepted as "visible, honest, strictly less harmful than the drop."

5.2 Verdict: EVOLVE — and it's already increment 0

Not eliminate: removing it reopens both August incidents (the 30-day replay shows it catching exactly those two real-person turns, 0 unexplained false positives). Not refactor-now: it is pure, measured, registered in the replay harness, and additive. The architecture source of truth's honesty-layer migration plan names increment 0 as "fix the outbound-composition contract that let a correct answer get silently dropped across tool-calling rounds" — #6278 is that increment, for the mechanical-token subset. Evolution path: (a) short term, keep as-is; (b) when the policy gate registry is armed and vocabularies unified (§6), re-home it as a rewrite-class policy so its verdict/trace/measurement ride the same rails; (c) the residual omission classes (dates, unit numbers, answers) belong to the judge's future completeness check + optional bounded regenerate (§3.2), not to more regexes — piling fact-class regexes onto the salvage is how guard sprawl started.

§6 Recommendation — arm and converge, don't rebuild

Is a refactor warranted? Yes — but it is an arming and consolidation refactor, not a build. The settled verdicts in the architecture source of truth (§4.5) already prescribe the single-chokepoint, three-way-verdict design; the code already implements it; what's missing is the operational sequence that turns it on without repeating P8 (the Aug 13–16 guard that told three real customers "nothing is scheduled" — reverted, and the reason the guard-change merge gate exists).

6.1 Can the Policy Gate express each existing guard? The fit table

Checked per guard against the gate's verdict shape (pass/repair/hold/block) — with one honest finding: the two registries speak different verdict vocabularies. outbound-safety.ts says pass/rewrite/block (text substitution); policy-gate.ts says pass/repair/hold/block (no rewrite/substitution verdict). Consolidation needs the union — add a rewrite (substitute/merge text) verdict to the gate contract, or the fair-housing/promise-backing/turnover-rewrite class cannot migrate.

Bottom line: every judgment-type guard fits once rewrite is added to the verdict contract; the strips and the reply-suppression layer correctly stay outside. Nothing found that structurally cannot migrate.

6.2 The recommendation in five lines

1. Write the missing ADR. The guard framework's change-control is cited as "ADR-0123" in code — that number is actually scheduled-job-placement; the guard-framework ADR was never written. Write it: one chokepoint, the unified verdict vocabulary, the measurement obligation, the arming ladder. (Fix the two stale citations in the same PR.)
2. Close the judge's two input gaps (inbound message; [[WHY]]/reasoning proxy) and add the completeness check the Daniel class needs — offline first.
3. Measure, then arm the Policy Gate: run the already-spec'd offline replay (60 days, all channels, subscription runners), hand-label per the RCA gate, then POLICY_GATE_MODE=enforce — repair + hold only, block stays reserved — bench property first, then Camellia. Same ladder for LEASING_FEE_FABRICATION_GUARD_MODE.
4. Migrate bolted-on guards into the registry one at a time, each with a zero-verdict-drift replay (same parity discipline the workflow-permutation harness uses). No behavior change per migration PR.
5. Decide the rewrite loop with data (§3.2): only after measurement shows the omission-residual class, as a bounded revise verdict — never as a general regenerate.

§7 Staged migration sketch — each stage with its proof gate

§8 Decisions — both answered

Two decisions from this investigation are raised on the Decisions tab: (1) arm the Policy Gate as the centralized layer on the staged ladder above, vs keep guards distributed; (2) build the push-back-to-Clara rewrite loop now, vs auto-heal + hold with revise-later. They compose with the four honesty-layer decisions already open on the architecture source of truth (H1 judge trigger frequency, H2 tool-execution gating, H3 voice mid-call vs post-call, H4 the credentialed nightly replay job) and Q1 (post-hoc judging paused) — this doc deliberately does not re-litigate those; stage 3 above assumes H1's recommended narrow-trigger start, and stage 5 is Q1's option C in miniature.

Both were answered by Gera on 2026-09-01, each matching the recommendation above.

1. Arm the Policy Gate — staged. "Arm it staged: enrich the judge inputs (current inbound + reasoning), run a 60-day offline replay with hand-labeling, then enforce (repair+hold only, block reserved) on the Willows test bench, then Camellia." The §7 ladder above is therefore the adopted plan, not a proposal. Note the ordering is load-bearing rather than cautious: per §3.3 the judge cannot see an omission without the inbound message, so input enrichment is a precondition for the replay to measure anything, not merely the gentlest first step. Nothing here is built yet.
2. No rewrite loop now. "No rewrite loop now: deterministic auto-heal + hold cover both August incidents; revisit once the armed gate’s measurement shows an omission class only a rewrite fixes." This settles the position stated in §3.2 and keeps row 6 of the §7 ladder conditional. The revisit trigger is explicit and belongs to stage 2’s replay: an omission class that repair, salvage and hold demonstrably do not cover. Absent that evidence, the revise verdict does not get built.

The four honesty-layer decisions on the architecture source of truth (H1–H4) and Q1 remain open and are unaffected by these two answers.

Appendix — verification notes, and corrections to the pre-investigation scoping

• Confirmed: single ordered chokepoint; guard order 1→8 as scoped; policy gate disarmed-by-default with full enforce wiring; the spec docs (policy-gate-spec.md, source-of-truth §4.5) matching Fede's verdict vocabulary; "push back to rewrite" absent; the ADR-0123 miscitation (in outbound-safety.ts:59 and fair-housing-policy.ts:13; docs/adr/0123-scheduled-job-placement.md is unrelated — no guard-framework ADR exists).
• Corrections to the scoping pass: (a) it undercounted the chokepoint — it missed the observe-only output gate, the Anomaly Review Gate, the three turnover honesty guards, team-inbox silence, the same-turn escalation constraint, and a second disarmed guard (leasing fee-fabrication) — 16 mechanisms, not 8; (b) line numbers drifted post-#6278 (salvage call site CM L8493, policy gate L9610, not ~L183/~L9565); (c) "neither repair nor hold loops back to the model" is true, but fair housing's rewrite is LLM-authored (classifier-side), and FirstTurnViolation is a real bounded regenerate — the missing thing is precisely a guard-verdict-driven regenerate; (d) the claim that the armed gate would have "caught this at the centralized layer" is wrong for the specced judge — the Daniel failure is an omission, which the claim-verifying judge cannot see without the inbound message input (§3.3). That nuance materially changes what "arm it" buys.
• Verified from prod (read-only): the incident trace's 10 steps (§1.2). No prod writes were made by this investigation.
• Not verified / stated plainly: whether POLICY_GATE_MODE / LEASING_FEE_FABRICATION_GUARD_MODE are set in the Vercel prod env (code defaults are off and fail-closed; no evidence anywhere of an arming flip — the spec, PR #5815 and the code comments all describe them as never armed). The com.propflow.eval-pipeline launchd finding is for the mini only; if the job lives on another machine it may still run, but evals/candidates/ is absent from the repo either way. Guard latency figures (the ~25s fair-housing note) are taken from code comments, not re-measured.