Superseded as the living doc → the Eval & Testing Roadmap (this page stays as the frozen record, 2026-08-20).

Proposed — pending Fede's review · researched 2026-08-20 (8-agent context fleet over the repo + practice)

The Quality System Plan

One pipeline for measuring Clara, one CI check that tells a Claude builder they made things worse, and a rebuilt outbound checkpoint that sees the whole trace and can send / repair / hold. Sequenced for the next five weeks, ending in decisions made on measured numbers.

In one paragraph. We own many measuring instruments — the full census is the eval inventory (2026-08-20): ten separate testing systems, including 56 behavior suites, ~85 one-off eval scripts across two overlapping folders (one not even in git), 275 incident-pinning drift tests, a 140-scenario voice harness, the 385-label golden set, and 30+ copy-pasted judge implementations — but no single system: different graders, different corpora, no signal to a PR that it made Clara worse, and the outbound guard sees the least context of any component while carrying the most responsibility. This plan unifies measurement into one pipeline, gives CI a regression voice that Claude builders act on, and rebuilds the outbound checkpoint around the full trace and a truth ledger with a three-way disposition: send, repair, or hold for an internal human. It is sequenced for the next five weeks and ends in decisions made on measured numbers, not vibes.

1. The three layers, tonight's evidence
2. One measurement pipeline
3. CI that tells Claude it made things worse
4. Input layer — context receipts & truth diff
5. Outbound checkpoint v2
6. The hold queue
6b. Post-hoc prod judging — the daily reviewer, paused
7. Sequencing — the next five weeks
7b. The three-layer POC — complete, results
8. The decisions
9. What we are NOT doing
Method & caveats

TL;DR

2 pipelinestext and voice build Clara's context in two separate, independently-maintained code trees — a fix to one does not reach the other

5 minhow stale application status can be before the next AppFolio sync — the named mechanism behind wrong applicant context

6,374 linesthe single function that runs the whole text-channel agentic loop, guards included

20 messagesthe trailing window every live outbound guard sees — never the full conversation, never the full tool history

0 jobstoday, nothing runs the current code and a proposed change side by side on the same conversations to see which one does worse — every automated check compares new output against an old saved answer sheet

OFFthe outbound safety check is built and merged but switched off everywhere in prod — founder decision, stays off for now

Five weeks of work below: one measurement pipeline, a paired CI regression job that reports into the same review loop Claude builders already act on, structured context receipts with staleness disclosed, and an outbound checkpoint extension proven offline before any live step — six decisions gate what happens after.

1. The three layers, with tonight's evidence

The founder's taxonomy — INPUT (wrong context reaches Clara), LOOP (Clara reasons/calls tools wrong), OUTBOUND (a bad reply almost ships) — maps cleanly onto tonight's fleet findings.

INPUT

Two separate context pipelines, not one. Email/SMS/Telegram turns resolve ~10 context objects and pass them as 16 frozen positional arguments into getUnifiedSystemPrompt() in clara-unified.ts. Voice runs an entirely separate 2,109-line reimplementation in the personalization route that never calls it. agents/clara/lib/agent/conversation-manager.ts:6365 + clara-unified.ts:732-900 (16-arg signature); src/app/api/voice/personalization/route.ts:1-75, 1798-1830
The 16-arg signature is already frozen — the code's own comment says a 17th context fact needs an options-object migration first, i.e. the team already knows this design is near its ceiling. conversation-manager.ts ~5770
Application status has three representations that can disagree — a funnel stage, the raw PMS status string, and a household-level fallback — read together because stage alone can't distinguish "pending" from "unrecognized." agents/clara/lib/agent/application-status-scope.ts:33-56; conversation-manager.ts:5729-5738
The AppFolio rental-applications poller runs every 5 minutes — the module's own header states our row is "a SYNC of AppFolio's truth, not the truth itself," so status can be stale for unbounded minutes between a real decision and our next sync. This is the named mechanism behind "wrong applicant status is common." lambda/appfolio-sync/handler.ts:11 (rate(5 minutes)); application-status-scope.ts:88-97
The status mapper collapses every unrecognized PMS status into the same bucket as pending — only approved/denied/withdrawn are special-cased; any other status string, including a new one AppFolio adds upstream, silently lands in APPLIED with no error. src/lib/domain/pms/writers/rental-application.ts:345-351
The allowlist-writer trap. The DynamoDB writer for prospect records is a deliberate SET-field allowlist — a field added to the TYPE is not persisted until it's added to the allowlist too, and nothing fails. This exact bug shipped for householdId: households were created correctly but never became reachable from a prospect row, silently, until a paginated production scan caught it. src/lib/data/dynamo/inquiries.ts:483-501 ("THIS WRITER IS AN ALLOWLIST, AND THAT IS THE TRAP"); fix in commit f60f1e5790, 2026-08-04, PR #5408
A removed identity-merge heuristic once fused two strangers. An inbound SMS from an unrecognized number used to get attached to whichever prospect at that property had a pending tour and no phone claim — when wrong, it merged two people's tour, notes, and consent record into one identity with no way to detect or undo it. Removed 2026-08-09: merges now require an exact verified claim or an explicit human action. commit 8b1a5076dc, 2026-08-09

LOOP

Hallucinations and wrong or missing tool calls are the two failure shapes.
The entire text-channel agentic loop, guards included, lives in one 6,374-line function — handleIncomingMessage, lines 3089 to the file's end at 9462, confirmed no other top-level declaration appears in between. agents/clara/lib/agent/conversation-manager.ts:3089-9462
Guards today see only the current turn's own tool steps plus a trailing 20-message window — explicitly "only earlier turns are history," capped, never the full thread — and a hand-picked grounding struct of four facts, not the actual system prompt text Clara was given. conversation-manager.ts:8786-8790 (20-message slice); src/lib/domain/compliance/policy-gate-types.ts:106-137

OUTBOUND — post-P8 reality

What's live today is a set of narrow regex/deterministic checks — the hallucination guard (renewal $/term, action-followthrough, tour-time-followthrough), two loop breakers (runaway-ceiling, farewell-loop), three turnover honesty guards, the same-turn escalation constraint, and the promise-backing "make it true" policy — plus one LLM classifier, fair-housing. conversation-manager.ts:3173, 3265, 7982-8798 (guard pipeline order)
The policy gate (PR #5815) is merged and real — a genuine checkpoint with pass/hold/repair/block verdicts and a typed PolicyGateGrounding — but OFF: POLICY_GATE_MODE defaults off and is set nowhere in prod config. Founder decision: stays off, no arming, no shadow, for now. src/lib/domain/compliance/policy-gate.ts:103-112; docs/planning/policy-gate-spec.md
The P8 lesson. A whole-reply canned-replacement guard ("never fabricate confirmations or slots") went live 2026-08-14 and fired on two real Camellia customers with zero true positives — a deterministic replay of 177 real turns found 28 (15.8%) would have been altered, none of them actual fabrications. Full revert the next day; the redesign doctrine since is replay-before-live, never simulated. PR #5791, merged 2026-08-15T20:31:44Z; docs/planning/policy-gate-spec.md

Design principle for this whole plan: layer attribution by construction, not by judge. Research shows post-hoc attribution is immature — the best benchmarked method names the responsible step correctly only 14.2% of the time (the Who&When benchmark, 127 annotated multi-agent failure logs; even frontier reasoning models don't reach practical usability). So each layer writes its own receipt — a context receipt, tool results, a guard verdict — and a failure's layer is read off the receipts deterministically, never guessed by a judge reading the whole trace after the fact. arxiv.org/abs/2505.00212 ("Who&When")

2. One measurement pipeline ("one ruler")

One case format for everything. The golden set (385 cases), the would-flag bed, suite cases, and future distilled desk verdicts converge on one corpus format: input, expected behavior, LAYER tag, lane, provenance, and a fail-before proof reference. The repo's own doctrine already mandates fail-before evidence — "Every new eval case must fail on the pre-change code. A case that passes on both sides pins nothing" — this format is what records it. CLAUDE.md "Proof ladder" (Fede, 2026-08-12); .github/reviewer-lenses/evals-and-tests.md [M-9]
One grader library — but honestly staged. The judges live on opposite sides of a live build boundary: the trajectory judge sits in the agent-side code tree, the calibration harness in the app-side tree, and the same import alias resolves to a different directory in each — with a bundler fallback plugin currently papering over that seam mid-migration. So v1 is a thin re-export shim per tree (same rubric, same pinned judge, two one-line entry points), and true single-module unification is scheduled with the module-independence work already in progress — not squeezed into week 1. The prod judges, the calibration harness, and promptfoo's rubric suites all consume the same rubric either way — and the inventory counts 30+ copy-pasted judge implementations today, so every suite that adopts the shared module retires a private copy. /a/eval-inventory-2026-08-20 Judge model pinned, rubric versions bumped on change, judge temperature near zero, repeats reserved for borderline/high-impact cases — regressions are read off smoothed trends, never a single flaky run. agents/clara/lib/agent/guards/trajectory-judge.ts:19-25 (the existing judge, fail-open on timeout by design); research: openlayer.com/blog/llm-as-judge-evaluation-guide
One runner tree. The inventory found ~85 one-off eval scripts split across two overlapping folders — one of which is not in git at all. Untracked scripts cannot count as merge evidence: everything moves into the tracked tree, reviewed, and the duplicate folder dies. /a/eval-inventory-2026-08-20 (move 2)
The weekly scoreboard: green rate by lane, judge-human agreement, and coverage — in-house, per the founders' 2026-08-18 decision that quality scoring stays in-house rather than a vendor platform.

3. CI that tells Claude it made things worse

The missing primitive, precisely. No harness today runs merge-base code and head code live, in the same job — every subscription-powered eval runner in the repo is in-process on the current checkout only, comparing model tiers or comparing head output against a static file. Existing "diffs" are one live side vs. one checked-in golden file (--diff-golden in pipeline-lab-evals; trace-replay comparing a stored production trace against a checked-in baseline). scripts/lib/subscription-eval-runner.ts:1-40; .github/workflows/intent-evals.yml:143-160 (trace-replay)
Build the paired replay job — and treat it as new plumbing, because it is. Nothing like it exists in the repo today, so week 2 is a proving spike on ONE small fast lane (intent), not a full rollout. Mechanism, named up front: a sibling git-worktree checkout of the merge-base with its own dependency install — correct even when the change being tested moves files around (cheaper per-file extraction breaks on exactly the changes this job must catch). The spike must answer four known traps before anything gates: full git history is needed to compute the merge-base; both sides are graded by the newer harness, and if the older side's code can't be driven by it the job reports "incomparable" honestly instead of inventing a delta; PRs from forks don't get the login credential, so the job skips there with a visible notice (the repo's existing publish step already does this); and the run costs roughly double a single-sided eval — measured in the spike BEFORE the old daily cost cap is retired, not after. (Unrelated to the held "corpus re-run gate" PR — same words, different system; that PR stays held.) Same corpus slice, same pinned judge, and the same credential path on both sides — a measured drift already exists between the subscription bearer and a metered key on at least one input class (a 5%-gated corpus drifted to 6.7% from credential choice alone), so the two legs of any comparison must never mix paths. .github/workflows/intent-evals.yml:99-142 (the 6.7% credential drift, reproduced via a dedicated 2x2 A/B script)
CI already runs evals on the subscription OAuth token — a drift test positively pins six workflows (nightly, renewal-evals, renewal-evals-sms, renewal-evals-scoping, intent-evals, clara-behavior-evals) to the OAuth bearer, never the metered key, following a 2026-07-29 incident where five PR-triggered workflows were silently spending the production metered key. The paired job follows the same, now-mandatory pattern. src/__tests__/no-metered-key-in-workflows.drift.test.ts:1-63; commit 46e8e68744, PR #5070
Reporting: one sticky PR comment with an HTML marker — the exact pattern ui-verify (Playwright against the PR's live preview) and pipeline-lab-evals (a per-preset pass/fail table, already upserted via the same find-or-create pattern) already use — carrying a per-case delta table: cases flipped fail→pass and pass→fail, each with its LAYER tag and a one-command repro. .github/workflows/ci.yml:3296-3307 (ui-verify marker); .github/workflows/pipeline-lab-evals.yml:487-552 (per-preset table)
The review bot already reads sticky marker comments as evidence — its turn-1 prompt is explicitly told to fetch and cite the ui-verify comment as "rendered evidence the diff alone cannot give you." A new  marker plus one mirrored paragraph is the minimal-diff way to put a regression report into the same loop that already drives Claude builders to fix review findings — this is the answer to "CI gives Claude feedback." .github/workflows/claude-code-review.yml:1293-1301
Make the existing prompt-change gate real. The check that says "a prompt change needs an eval diff" is advisory today — it sits outside branch protection and its escape hatch needs no sign-off. It becomes an actually-required check in the same step the paired job goes blocking, under the q-gate-lanes decision below. /a/eval-inventory-2026-08-20 (move 3)
Rollout: advisory for one week on the reliable fast lanes (renewals, intent) → blocking, per the open gating decision below. The umbrella sweep's 1-in-~200 execution budget (a daily cost cap that already let ~13 failing domains sit unnoticed on main for weeks behind a false-green skip) gets retired in favor of path-filtered fast lanes plus the nightly full sweep. A nightly regression on main posts to #alerts and opens a card, the same paging mechanism the anomaly review gate already uses. .github/workflows/clara-behavior-evals.yml (daily-cap budget job); src/lib/platform/observability/notify.ts:1-45
Statistical honesty. A paired McNemar-style resolution check runs alongside every delta: if the observed gap needs more samples than the slice actually has to detect it at target power, the comment says "underpowered" — never a false verdict. On real leaderboards, naive power calculators underestimated the needed sample size by roughly 50%, and 11 of 40 adjacent-rank comparisons in one study lacked enough resolution to trust the ranking. arxiv.org/html/2605.30315v1 ("resolution diagnostics" for paired LLM eval comparisons)

4. Input layer — context receipts and the truth diff

Context receipt per turn. The ~10 context objects already assembled in the loop (who Clara believes the sender is, funnel stage, application status + its raw PMS string + last-synced timestamp, tour state, renewal state, specials in effect, escalation state) get logged as a compact structured receipt on the AgentTrace. Today AgentTrace has no fields for any of this — only an optional, togglable raw prompt-text dump on 3 of 4 channels, and none at all for voice. src/lib/data/types.ts:7546-7568 (AgentTraceStepType — no identity/context step type exists), :7705-7750 (AgentTrace interface); conversation-manager.ts:6461, 6712 (capturePromptForTrace)
Voice writes the same receipt from its own pipeline — right now the personalization route builds its dynamic_variables and returns them directly to ElevenLabs with zero first-party record of what Clara was actually told at call start. confirmed by grep of src/app/api/voice/personalization/route.ts — no AgentTrace/saveAgentTrace usage anywhere in the file
Staleness is part of the receipt, not hidden from it. Each field carries its source and last-sync stamp, so the 5-minute application-status poller lag becomes visible and disclosable instead of invisible. application-status-scope.ts:88-97
Deterministic input alarms: the status mapper's catch-all bucket alerts on an unrecognized PMS status instead of silently mapping it to applied; an allowlist drift test generalizes the householdId lesson — a field present in the TypeScript type but absent from the writer's allowlist fails CI. src/lib/domain/pms/writers/rental-application.ts:345-351; src/lib/data/dynamo/inquiries.ts:483-501
Nightly truth diff. Sample recent turns, compare each receipt against the real sources of truth as-of-now; mismatches become Quality Desk items with reason "wrong context" — a new, countable defect class with a weekly number. This mirrors the "context quality testing" pattern research describes as a distinct QA layer upstream of output evaluation, scoring the assembled context itself (freshness, source certification, correct term used) rather than only the final reply. atlan.com/know/ai-agent/context-quality-testing-for-ai-agents/
Per-property nightly checks, not Willows-only. Every live harness today is hard-locked to the Willows test bench — which is exactly why nothing automated could see a real-property incident. The nightly Camellia containment check now being built is the first instance of the fix; the pattern generalizes here: every real property gets a nightly "nothing leaked, context matches truth" pass reporting into the same scoreboard. /a/eval-inventory-2026-08-20 (move 4)
Prompt-conflict lint v1 (deterministic only). Extend the existing 275-file .drift.test.ts pattern — each pins ONE specific invariant discovered after a bug, none scans for a NEW contradiction — to structural conflict classes already burned into the history: the duplicate LANGUAGE: block that shipped with contradictory instructions and "recency, but it's a coin-flip" as the only resolution mechanism; the denied-applicant override that got lost when placed mid-prompt instead of at the very end of a ~117k-character prompt; ad-hoc override sentences with no registered precedence marker. commit 85c53edb99, 2026-05-05 (the LANGUAGE: dedupe, "recency typically wins but it's a coin-flip"); PR #5631, merged 2026-08-11 (denied-applicant repositioning fix)
Semantic conflict detection is explicitly deferred — research-grade, not a five-week item. The Arbiter framework (block-decompose a prompt, then pairwise-check for formal contradiction) and WIRE (SAT-solve for symbolic rule conflicts, then generate a live "witness" scenario that actually triggers both rules — only 35.4% of jointly-governed scenarios in one study resolved in compliance with both) are the right shape for a future version. The unmerged checkInjectable(principle) seam — a pure, fail-closed-by-construction gate for whether a candidate prompt block may join the assembled prompt — is where it would eventually live. src/lib/domain/grading/injection.ts (unmerged, PR #5786); arxiv.org/pdf/2603.08993 (Arbiter); arxiv.org/html/2605.27784v1 (WIRE, 35.4% figure)

5. Outbound checkpoint v2 — full context, grounded, repair-not-rewrite

Do not build a new guard — extend the outbound safety check that is already built and merged but switched off. Three extensions: PR #5815; POLICY_GATE_MODE

(1) Grounding → truth ledger

Ledger v1, honestly scoped: wrap the 4 fields the merged-but-off safety check already carries (tourSnapshot, preparedOffer, currentRent, conversationEscalated — the only consolidation seam that exists today) behind one named accessor, redirect the renewal read through the canonical renewal accessor it currently bypasses, and stamp every field with its source and last-sync time. The remaining rows below are v2, each its own scoped change — the work-order aggregate is a state change the code itself says wants its own PR, and the escalation-matter and specials rows need new wiring (no guard reads them today), so billing them as "consolidation" would overstate what exists. The table is the full inventory with known holes:

Ledger row	Freshness	Known hole
Tour snapshot	Immediate — written synchronously by the tour tool handlers, no sync lag	The guard reads a single denormalized pointer on the prospect row; the prompt reads across every tour the person owns on any channel — they can, by design, see two different tours for the same person
preparedOffer / currentRent	~60s poller for preparedOffer; currentRent depends on a manual rent-roll upload — could be stale for weeks	A separate live-scraped pricing system (UnitPrice) is never written back onto Unit.rent — two "current rent" surfaces coexist unreconciled
Work-order status	~1-minute PMS sync — the fastest cadence of any job	No durable "current turnover state" aggregate exists; guards ground only against this turn's own tool outcome, so an honest failure can be wrongly re-blocked next turn with no memory of the prior one
Renewal saga state	Single-writer, optimistic-version-locked — the best-governed object in this inventory	A canonical read-side accessor already exists (renewal-view.ts) but the guard's own snapshot assembly bypasses it today, a duplicate read path
Application status + staleness stamp	5-minute poller — the known weak spot	An unbounded-staleness hole: once an application ages out of the PMS report window, the sync stops seeing the row and a stale status can persist indefinitely with nothing bounding its age
Specials, with source tags	Overwritten on every scraper watcher run unless manually entered	Dual-writer risk — no per-entry freshness timestamp beyond the last write's origin tag
Escalation matter state	Live, per-question, durable — the modern system	Not read by the guard/policy-gate grounding object today; the guard still reads a coarser boolean that its own doc calls a weak signal ("auto-reopens on any inbound message, proves nothing about the team acting")

Structural risk to respect: core state types are declared twice in this repo (once under agents/, once under src/), a deliberate parallel tree that must be kept manually in sync — the ledger needs to live where both trees consume it. agents/clara/lib/data/types.ts:1-5 ("PARALLEL TREE — read this before adding or modifying types here")

(2) Full trace input, structured

The checkpoint sees the inbound message, the context receipt (section 4), the tool calls and their actual results, and the draft — as structured fields, not prose. OpenAI's own reference agent framework documents that its standard output guardrail sees only the final draft reply — not the question, the retrieval, or which tools ran — a real, named gap. And a 2026 benchmark built specifically for multi-step tool-calling guardrails found accuracy was driven more by whether the guard could correctly PARSE the structured trace than by its actual safety judgment — a "structural bottleneck." Keep the input schema boring. Scope note: checkpoint v2 stays turn-scoped — this turn's inbound, receipt, tools, and draft, on top of the existing trailing conversation window — full-conversation review remains the separate async reviewer's job, so this widening adds no new context-length cost. openai.github.io/openai-agents-python/guardrails/; arxiv.org/abs/2604.07223 (TraceSafe)

(3) Why-Clara-decided

The reply-rationale trace (ADR-0110's message-rationale machinery, already pinned by its own promptfoo suite) rides along so the judge sees Clara's own stated grounds for the reply, not just the reply text.

Repair, never rewrite

Span-level only: a claim that fails grounding is repaired to the ledger's truth or removed, each repair citing its ledger row; repaired drafts re-run the full check; no canned replacements, no whole-reply regeneration — P8's exact failure mode. Research backs this precisely: blind full regeneration broke 47 previously-correct answers out of ~1,300 test problems in one study while a version that only swaps in a fix when a separate checker can prove it's better broke close to zero. The repo's own live pattern is the template: the promise-backing "make it true" fixpoint rewrite and the escalation-claims repair, both grounded on the same shared detector, hold a combined zero false positives across a 2,594-turn replay and a 30-day, 515-conversation live production check. arxiv.org/abs/2605.24613 ("Guarded Repair"); src/lib/domain/compliance/policies/promise-backing-policy.ts:1-194 (PR #5915); agents/clara/lib/agent/policies/escalation-claims-policy.ts:1-52

Three-way disposition

PASS (send as-is) / REPAIR (send repaired, audit-marked — the live blockedDraft registry, already merged and covering 7 guard families, already records before/after) / HOLD (do not send; page a human). Confidence is one input weighted ≤~15% alongside deterministic rule hits and grounding results — production guidance converges on never trusting a model's self-reported confidence alone. Protected categories and legal/numbers lanes bias to hold; fail-closed on checker error for action-claiming or legal-adjacent drafts, fail-open only for low-stakes conversational lanes — the per-lane table is a founder decision below. src/lib/domain/messaging/blocked-draft.ts:1-81 (BLOCKED_DRAFT_GUARD_REGISTRY, PR #5764); confident-ai.com/blog/human-in-the-loop-ai-agent-evaluation (≤15% confidence weighting)

The false-positive law. Block-rate and false-block-rate correlate at ~0.878 in benchmark analysis — a filter made stricter to catch more real problems also blocks more good replies. Stacking five 90%-accurate checks in a row already means a legitimate reply has only about a 59% chance of getting through untouched, and once a guard's wrong-block rate passes roughly 2%, the trust it costs outweighs the safety it buys. Every widening of scope must re-prove precision on replay before it ships — already repo doctrine post-P8; this plan makes it the permanent rule for the checkpoint. tianpan.co/blog/2026-04-20-alignment-tax-product-ai-safety-guardrails

Voice: post-call reconciliation against the same ledger only — no inline latency, the standing founder rule. This matches converging practitioner guidance that guard checks belong async/post-call on anything latency-sensitive; a lean three-check guard stack alone measured 350-400ms.

6. The hold queue (human-in-the-loop for internal engineering)

Today "hold" is a misnomer. The anomaly review gate substitutes a fact-free holding sentence and SENDS it in the same turn, then separately pages a human — nothing ever sits genuinely unsent. conversation-manager.ts:8334 (`responseText = gate.holdingMessage ?? '';` executes synchronously, no wait-for-human step)
The two true precedents — the only genuinely held-until-approved outbound in the whole repo — are /review's collections_dun (a composed past-due reminder held for approval; approving it is what sends it) and vendor_outreach. Each has its own decision route that is the ONLY thing that dispatches the message. src/lib/domain/leasing/collections/review-rows.ts:1-14; src/lib/domain/tenant-confirmation/review-rows.ts:81-96
Build a new held_outbound ReviewType on the live /review queue, following the collections_dun pattern (approve = send, edit-then-send, reject + reason) — the queue already merges four item types this same way, with approve/reject actions, a detail drawer, and 5-second polling. src/app/(workspace)/(operations)/review/page.tsx:1-14, 83, 451
A new platform_admin-only visibility gate — today /review is visible to a property's own PM/leasing/accounting staff, not just PropFlow engineering. One existing item type's docstring already claims "held for internal engineering review" while its actual auth check is the same broad one — a genuine, previously-unflagged gap this plan closes rather than inherits. src/app/(workspace)/(operations)/review/layout.tsx:1-21; src/app/api/renewal-change-reviews/route.ts:28-53
notify() pages #alerts on every hold, the same fire-and-forget Slack path the anomaly review gate already uses. SLA clock with a fail-safe: if a hold ages past the lane's SLA, the fallback is the honest escalated-ack template already used by the same-turn escalation constraint — never silence, never the unchecked draft. src/lib/platform/observability/notify.ts:1-45; conversation-manager.ts:1280-1345 (decideSameTurnEscalationOutbound / ESCALATED_ACK_TEMPLATES)
Flywheel without the held PRs. Every hold outcome — approved, edited, rejected — is auto-captured as a labeled corpus case; reviewer-overturned cases recalibrate the checkpoint. Production guidance converges on exactly this loop: every case a human reviewer overturns should both recalibrate the checker and become a regression test. This gets corpus growth from human review without touching the four held promotion-pipeline PRs, which stay held per the founder's decision until the quality bar (section 8) is met. langchain.com/resources/llm-as-a-judge
Audit surface: revive the read-side blocked-draft UI (branch fede/blocked-draft-ui, 3 commits, stale — 202 commits behind main, no PR ever opened) so anyone can see "guard rewrote/held this, view both versions" inline in the thread. git log origin/fede/blocked-draft-ui — 3543d91f, 65570d6f, a069bfc7, all 2026-08-14; `gh pr list --head fede/blocked-draft-ui --state all` = []

6b. Post-hoc prod judging — the daily reviewer, paused Proposed — pending review

Every layer above checks a conversation before or during the send. This section is about the fourth check: something reading a whole conversation after it's over and grading whether Clara handled it well — the only layer that can catch a mistake none of the live guards were built to look for.

Who checks a conversation, and when

CI replay gate
before a code change ships
proven on a small surface

→

Policy gate
at send, turn by turn
built & merged, switched OFF

→

Daily reviewer
the day's conversations, after the fact
PAUSED since ~Aug 16

The gap sits at the third box. Nothing today reads a finished conversation and rules on whether Clara got it right — the two live checks above only ever see one code change or one turn at a time, never the whole story after it's done.

What we found (verified 2026-08-21/22, repo-side and Slack-side). The app already has a daily reviewer built for exactly this: conversation-review-daily (merged 2026-07-18). It runs on a schedule every afternoon, reads every conversation from the previous 24 hours in full — the transcript, the tool calls, the call analysis — and has an AI reviewer propose findings that must quote the exact message and cite it, then a second, independent AI checks and can strike down any finding that doesn't hold up. The result posts as a daily digest to the #agent-smith Slack channel. It was designed to run side by side with the team's existing daily list (Agent Smith) until it proved itself, then take over as the sole reviewer.

That handoff never happened. The schedule that fires the daily reviewer is paused — it last ran around August 16 and nobody restarted it. Separately, the team's plan for Agent Smith is to retire both daily lists (this reviewer and Agent Smith's own, which by design does nothing today) and replace them with a live feed of what's happening posted straight to #apartment-camellia as it happens. That feed is meant to eventually carry a 1–5 quality score per conversation — but a live feed of events is not the same thing as a judge; scoring needs a reviewer behind it, and today there isn't one running.

Net effect: since about August 16, nothing has reviewed a finished prod conversation and said whether Clara handled it well. Every live guard above still runs — this is a gap in the after-the-fact check, not in the in-conversation ones.

q-posthoc-next — What do we do about the paused daily reviewer?

The schedule is paused, not deleted — nobody should delete it before this is answered. Coordinate with the team planning the #apartment-camellia activity feed before touching it either way; that plan and this decision both claim the same territory.

A. Resume the paused schedule now, as-is — zero build, keeps the advisory-only Slack digest running while the bigger question is worked out. B. Wait for the #apartment-camellia event stream and add judging onto it later — accepts an open-ended stretch with no post-hoc judge at all. C. Resume the schedule now AND fold post-hoc judging into the quality system as a real stage: the same Cerberus engine already grading other layers writes a verdict on every conversation after the fact (async only — never inline on a live call, per the standing no-added-voice-latency rule), those verdicts feed a proposal queue into the grading desk / golden set for a human to accept or reject (nothing enters the golden set on its own), and the daily Slack digest keeps running until the event stream is ready to carry the rating itself. RECOMMENDED

In plain terms: right now nobody checks yesterday's conversations to see if Clara did a good job — the robot that used to do that got switched off about a week ago and nobody noticed until this week. Turning it back on costs nothing. Option C also makes its findings useful instead of just scrolling by in Slack: they become suggested teaching examples that a person still has to approve.

q-posthoc-desk — Once verdicts exist, what happens to them?

Only asked if C is chosen above.

1. Stay advisory, Slack-only — read the digest, nothing writes into the desk. 2. Feed the grading desk as proposals a human confirms before they count — same human-in-the-loop pattern the rest of this plan uses everywhere else. RECOMMENDED 3. Auto-import once the verdict's confidence clears a bar — fastest growth, no human check on what enters the golden set.

7. Sequencing — the next five weeks

Week	Theme	Ships	Testbed
1 now – Aug 24 (a short week: ~3 working days)	One ruler	Case format + LAYER tags; corpus fold-in (golden 385 + would-flag bed + top suites); baseline scoreboard number. Grader unification is deliberately NOT here — it crosses a live build boundary mid-migration (section 2); week 1 ships only the per-tree shim, and only if it stays a one-liner each	None needed — offline
2 Aug 24 – 31	CI delta — prove the mechanism	Paired base-vs-head spike on ONE lane (intent): sibling merge-base checkout working end to end, the four known traps answered (history depth, harness-version mismatch → honest "incomparable", fork-PR credential skip, measured cost of doubling), advisory output only. Parallel, small: context receipt on AgentTrace + the two deterministic input alarms (catch-all mapper alert, allowlist drift test)	Intent corpus
3 Aug 31 – Sep 7	CI delta rollout + input truth	Paired job extends to renewals; sticky delta comment + the underpowered-honesty check; nightly main sweep → #alerts; umbrella budget retirement (only once the spike's cost number is in hand); nightly truth-diff job → desk "wrong context" items; golden set layer-labeling pass	Renewals corpus; Camellia traffic samples, read-only — no writes, no effect on anything sent
4 Sep 7 – 14	Ledger + checkpoint + hold queue	Truth ledger v1 (the honest 4-field scope above); checkpoint v2 trace-input widening; prompt-conflict lint v1; `held_outbound` queue built with the new engineering-only visibility gate	The Willows sandbox (synthetic)
5 Sep 14 – 21	Prove offline, drill the hold	Offline subscription replay of checkpoint v2 over real recent traffic (policy-gate-spec doctrine: every flagged case individually reviewed) → measured precision/recall published; end-to-end hold drill on the Willows bench (synthetic traffic only — a real hold, page, approve/edit/reject, corpus capture, no real customer); blocked-draft UI revived from its stale branch (a real rebase, 200+ commits behind main)	The Willows sandbox (synthetic)

Then: the decision gate (section 8). Nothing goes live on real traffic in these five weeks beyond receipts/logging — every live-behavior change waits for the decisions below. One more output of reviewing this page: the founders split ownership of the lanes above (almost everything in the inventory is single-author today) — the census to divide is the eval inventory; this page is where the split gets written down.

7b. The three-layer POC — COMPLETE, results published

Done 2026-08-20, same day: all four prototype lanes built, run, and independently re-verified against the five real Camellia failures. Full metrics, pros/cons per lane, the verified overfitting audit, and the CI regression design: Catching Clara's Mistakes — the POC results. Headline: every lane caught 5/5 with 0 false alarms and 2/2 regression kills (all independently reproduced); the audit's real finding is that structural truth-diff checks generalize and are solved today, while the semantic classes were only phrase-matched and belong to the AI-judge leg — the architecture verdict is compose (our checkers + promptfoo chassis + one shared before/after diff), not adopt a framework.

The founder's framing, 2026-08-20: single-number evals are shallow — like unit tests. Multi-turn suites are better, but neither says which layer broke. So before the five weeks above harden anything, a proof-of-concept is testing the three-layer idea on real life: five real Camellia conversations that actually failed, one per failure class, pulled from the founder-graded golden set and sanitized (fake names, fake numbers — the conversations and failures are verbatim otherwise).

Input: a current tenant told "you moved out" because a stale occupancy record fed the context.
Input: Clara auto-replied on a thread a human teammate had already taken over.
Loop: a tool call failed and Clara invented a policy ("application links go out after a tour") to cover the gap.
Outbound, hold class: internal system narration leaked into a customer reply — the founder's ruling on the real case: kill the response, page engineering.
Outbound, repair class: "the team will follow up" promised with nothing behind it — the class the promise-backing change now catches in production, so it doubles as proof the harness catches a known-fixed bug.

Competing prototypes, one frozen scorecard. Up to four independent agent teams each build a small harness that must catch those five failures, attribute each to its layer by evidence (a receipt diff or a paired clean-twin diff — research puts cold-read judges at roughly 14–25% accuracy on "which step broke," so cold reads don't count), stay quiet on clean bait, and kill seeded mutations (the same conversation minimally worsened — the "did a change make things worse" primitive). The stacks, chosen from a four-lane research sweep of open-source frameworks and how agent companies (Anthropic, Sierra, Intercom, Decagon, Cursor) actually test — with one framing correction from the founder (2026-08-20): the research initially dinged every guardrail framework for grounding "text against text you hand it, not a live database," but the truth source is just a delivery mechanism — the truth ledger serializes into text at check time, so that objection evaporates and gets tested empirically instead of decided on paper. Four lanes: a hand-rolled TypeScript receipts-and-checkpoint harness (borrowing the frameworks' best ideas: a repair-not-just-block verdict vocabulary, span-level fixes, typed pass/hold/repair/block findings); an existing guardrail framework fed serialized truth as its source text (Guardrails AI, whose sentence-level provenance mode flags exactly which sentence is unsupported — the head-to-head the founder's poke demands); promptfoo extended with trace-path assertions (already in the stack, native subscription login); and a Python eval framework (inspect-ai, subscription-credential native) bringing trajectory scoring and full-transcript logs out of the box. The frameworks' remaining real differences — verdict vocabulary, span-level repair, regression diffing — are now things the scorecard measures, not assumptions.

The deliverable is the architecture plus the measured results: per prototype — catch rate on the five real failures, layer-attribution correctness, false alarms, mutation-kill rate, verdict quality, cost and wall time, maintenance burden — with pros and cons, and a concrete recommendation for how CI uses the winner to tell a builder "this change made Clara worse." Everything runs offline on sanitized fixtures under the subscription; nothing merges, nothing touches live traffic; whatever wins feeds the week-2/3 CI delta lanes above rather than replacing them.

8. The decisions — deferred until there is working software

Founder ruling, 2026-08-20: "I need working software before decisions." Nothing below blocks any build work, and none of these questions is being asked right now. Weeks 1–3 and the POC proceed on the do-nothing-live defaults (everything stays offline; nothing arms; the four held PRs stay held). Each question below comes back to the founders only when the working software that informs it exists — the POC results for the framework-shaped calls, the paired-CI spike numbers for the gating calls, the offline replay numbers for the checkpoint calls. The list stays here so the questions are pre-registered before the numbers exist — that is what keeps the eventual answers honest.

Answers save for everyone who opens this page; the recommended option is listed first and marked — recommendations become real asks only once the software behind them runs.

q-gate-lanes — Which suites become blocking first?

Blocking CI gates on AI behavior are already authorized in principle (D3, 2026-08-18) — this is about which checks go first. This same open call already sits on the eval inspection page; deciding it here supersedes that instance with the full five-week sequencing attached.

Narrow-reliable-first: renewals + intent fast lanes, advisory for one week, then blocking. RECOMMENDED Umbrella-first-after-budget-fix: broadest coverage, but only once the 1-in-~200 execution cap is retired. Deterministic-only-first: pure assert-based checks, no LLM judge, expand to judged checks later.

q-hold-host — Where do held messages live?

The fleet's ranked recommendation: extend the live /review queue rather than build a new surface — it already has the approve/reject mechanics, four prior extensions of the exact same pattern, and live polling; it just needs new platform_admin-only gating.

/review + a new held_outbound type, with platform_admin-only gating. RECOMMENDED A new, dedicated engineering-only page — cleaner isolation, more to build. Slack-only approval flow — fastest to ship, no browsable audit queue.

q-checkpoint-next — After the offline replay proves precision, what is the first live step?

The pre-send gate stays OFF today per the founder's standing decision. This decision only pre-registers what evidence would justify the next step — nothing moves without his explicit yes, even on green replay numbers.

Stay offline another cycle — replay again on fresh conversations, don't touch live traffic yet. Matches the standing no-shadow decision. RECOMMENDED A bounded verdict-logging window on the Camellia SMS leasing lane — hard end date (e.g. 7 days), logs verdicts only, sends unchanged, switches itself off at the deadline. Deliberately NOT a standing "shadow mode": an always-on second path was already rejected in code review on the very change that built this safety check, and this plan does not reopen that. Choosing this still reopens the founder's no-shadow decision, so it needs an explicit sign-off that a time-boxed window is different from what was rejected. The same bounded window, but on renewals first — the best-governed state object, lowest-risk surface. Also reopens the no-shadow decision, same sign-off required.

Evidence for the leasing lane specifically: the leasing gate scorecard — full-population replay (45/45 conversations, 135 draws), 4/5 hand-read flags were true catches, cross-validated by a same-night production audit.

q-unlock-bar — The numeric bar that unlocks the four held promotion-pipeline PRs

Fede's decision on 2026-08-20: the four PRs (#5782, #5784, #5785, #5786 — distiller, corpus re-run gate, promotion arm, principle injection) stay held "until agentic quality and testing significantly improve." This turns that sentence into numbers.

A: judge-human agreement ≥85% on the current golden set AND 2 consecutive weeks of blocking CI with zero false blocks AND checkpoint replay precision ≥90%. RECOMMENDED B: same shape, looser bar — 80% agreement / 1 week / 85% replay precision. C: founder judgment call at the end of the five weeks, no fixed numbers.

q-repair-scope — What may the checkpoint repair automatically?

The live "make it true" pattern today only ever repairs a claim class where the repair tool's success and the claim's truth are the same event (an escalation claim, backed by actually calling the forwarding tool) — never an arbitrary fact.

Numbers/dates/escalation-claims only — the proven classes, matching what's live today. RECOMMENDED Any claim that fails grounding — wider scope, unproven at this width. Nothing — repair stays human-only; every repair becomes a hold instead.

q-input-fix-priority — Biggest input-layer fix to schedule right after the five weeks?

Three real candidates surfaced tonight, each already documented as a known, named gap in the code.

Application-status staleness — event-driven sync or webhook, plus the stale-stamp disclosure this plan's receipts already surface. RECOMMENDED Voice/text context-pipeline unification — closes the two-tree problem, largest single lift. Prompt options-object migration + conflict lint v2 — unfreezes the 16-arg ceiling.

9. What we are NOT doing (scope fences)

Not merging the four held promotion-pipeline PRs — decided; the bar is q-unlock-bar above.
Not arming any pre-send gate — decided; only offline replay plus a pre-registered next-step decision (q-checkpoint-next).
No inline voice checks — decided, standing founder rule; post-call reconciliation only.
No semantic prompt-conflict AI in v1 — research-grade (Arbiter/WIRE), deferred past these five weeks.
No vendor eval platform — D1 decided 2026-08-18, in-house.
No rewrite of the 6,374-line loop inside these five weeks — the checkpoint and receipts wrap it instead; decomposition stays on the architecture roadmap.

Method & caveats

8-agent context fleet, 2026-08-20 (5 repo dives + 2 web research + this synthesis) — ~893K tokens, 419 tool calls across input-context, prompt-conflicts, guard-internals, hitl-surfaces, truth-ledger, ci-feedback, research-guardrails, and research-layers. Every claim above traces to a file:line, PR, or commit cited inline, or to the two named research lanes. No customer name, phone number, or email appears anywhere on this page. Builds on the 41-agent eval system inspection (2026-08-19 — the 54-suite inventory, the Sonnet 5 story, the same open which-suites-gate-first call this page's q-gate-lanes decision now carries forward) and the honesty-layer deep inspection (2026-08-16 — the P8 postmortem and the redesign doctrine this plan's outbound section builds on), both linked from the architecture source of truth. Companion census: the eval inventory (2026-08-20), produced by a parallel session as this plan's ground-truth inventory — its five consolidation moves are folded into sections 2–4 above.