Clara Quality Loop — Team Guide
How Clara's quality machinery works, how to use it day to day, and what we learned building it. Written for Gera; useful for anyone touching Clara. Updated 2026-08-23.
The loop in one line
You grade a real conversation→
the grade becomes a permanent test→
a CI gate replays all tests on every PR→
a judge grades live conversations→
recurring failures become prompt principles
Human judgment enters once — a thumbs-up or thumbs-down on a real conversation — and the system turns it into standing protection. Nothing regresses silently; every mistake fixed once stays fixed for every property.
How to use it daily
1. Grade conversations (the main habit)
- Open any conversation and hit the Quality Desk:
propflowai.co/clara/playground?item=<conversationId>, or right-click a specific message inside the conversation view to grade just that turn. - 👍 = Clara handled it right. 👎 = something wrong — you must pick a principle (the rule she broke). That's deliberate: it keeps the taxonomy meaningful and gives the pipeline something to build a test around.
- Verdicts are append-only and scoped (whole conversation, one message, or one step). Your prior verdicts show in the rail so you never re-grade blind.
- The Decision Line on each conversation shows the factory view of what happened: message in → context → reasoning → tools → which door the reply went out (delivered / guard caught it / human in loop). Use it to see why before grading.
2. What happens to your 👎 (automatic)
A nightly pipeline picks up new 👎 grades, drafts a regression test from the real conversation (identity replaced with a synthetic one, PII scrub fails closed, quotes anchored byte-exact), lands it in the shared test library, stamps a new version, and syncs it into the app. You get a one-line note in #alerts. No PRs to review, no approvals — the annotation was the approval.
Positive grades feed pass-tests too (locking in behavior we like), prioritized where the machine and human disagreed.
3. Adding or editing tests by hand
- The test corpus lives in its own repo:
PropFlow-Technologies/cerebrus. The app vendors it at a pinned tag + checksum underquality-gate-poc/. - Upstream first, always: change cases in cerebrus, tag, then run
scripts/sync-cerebrus.shin the app. Never editquality-gate-poc/fixtures/directly — CI's vendor-sync check will catch it, and hand edits get overwritten on the next sync. - App-specific deltas (like domain tags) live as documented patches applied on top of each sync — they survive upgrades; hand edits don't.
4. What guards your PRs
- Regression Gate — Leasing runs on every PR: it replays the corpus base-vs-head and only regressions go red — adding a new failing case never reddens someone else's PR. Currently advisory (not required to merge) while we build trust in it.
- The judge lane runs on subscription auth (claude-sonnet-4-6) — never metered API keys; a spend guard blocks those paths by design.
What's live vs. in flight
| Piece | Status | Notes |
|---|---|---|
| Quality Desk (grading UI) | LIVE | Rebuilt this week after Fede's first real session — see learnings below |
| Decision Line visualization | LIVE | On every conversation page |
| Regression gate (leasing) | LIVE, advisory | Runs every PR; not yet in branch protection |
| 👎 → test pipeline | MERGED, OFF | Awaiting arm + test-repo credentials; first supervised run planned |
| Live judge (real LLM grading) | BUILDING | Calibrating against ~385 human verdicts, 70/30 held-out split; today's placeholder is a loop-detector only |
| Principle ladder (grades → prompt rules) | PARKED | Four branches under test on a bench; nothing touches prod prompts |
| Voice safety | GAP | Decision pending — see below |
Findings from the first real annotation session
- The machine judges missed every mistake Fede caught. Not a model failure — the "judge" in production was a placeholder (a repeated-message detector plus two narrow checks). That's exactly why the calibrated real judge is the current build.
- Voice has zero safety pipeline. Texts and emails pass hallucination / promise-backing / fair-housing checks before sending; phone calls pass nothing, because the language model runs inside ElevenLabs and our code never sees the words before they're spoken. Two real incidents came from this (an unbacked "you're all set" and an invented fee).
- The voice lie is deterministic and preventable for free. Offline replay of the real incident: ambiguous tool wording → the model asserts the unbacked booking 20/20 times; honest, conflict-explicit tool wording → 0/20. Tool-side checking costs ~15 ms; a pre-speech model check costs ~846 ms. Full options + measurements: Voice Truth Guard — Decision (pending Fede's pick; no voice change merges without his explicit approval).
- Three production bugs surfaced and fixed: a tour-confirmation email firing before the tour was actually confirmed (now gated on the live workflow's confirmed state), Clara replying to our own notification cards (self-mail guard widened), and one person showing as two conversations because channels were split (one conversation per person, all channels).
- Vendoring discipline matters: the one time a test was added directly to the vendored copy, it broke the model — redone upstream-first. Patches are stored as diffs and fail closed on conflict, so a silent upstream revert can't launder itself through a sync.
- Annotation research (industry + our own): small, clean, principle-tagged golden sets beat big noisy ones; append-only verdicts preserve history; taxonomies grow as you go but every 👎 must land on a principle. The full playbook with the before/after UI: Grading & Annotation Guide.
Where things live
| Thing | Where |
|---|---|
| Grading UI | /clara/playground?item=… (or right-click a message in a conversation) |
| Test corpus | PropFlow-Technologies/cerebrus → vendored at quality-gate-poc/ |
| Live-grade seam | src/lib/domain/conversations/conversation-grade.ts (the judge slots in here) |
| Annotation playbook | grading-annotation-guide |
| Voice guard decision | voice-hallucination-guard-decision |
| Roadmap / architecture | eval-testing-roadmap · architecture-source-of-truth |
Ground rules that apply to everything above: evals run on the Claude subscription, never metered API keys · no voice-path merge without Fede's explicit approval · test identities are always synthetic, PII scrubbing fails closed.