Quality Loop — Proven on Leasing

1 · A human grades real conversations DONE — Fede's session

Fede's first real annotation session graded 95 conversations (179 verdicts total in production). Every 👎 lands on a named principle. The session also surfaced 7 UI bugs (all fixed) and 3 production incidents (all root-caused and fixed).

2 · A grade becomes a permanent test — automatically PROVEN LIVE, SWITCH ON

The pipeline ran live and landed two of Fede's thumbs-downs as permanent tests, with zero human steps each: drafted (synthetic identity, PII scrub fails closed), machine-reviewed, landed in the shared library, version stamped (now v0.1.8), synced back into the app, announced in one plain line in #alerts. Duplicate grades of the same mistake were recognized and folded in. The gate went from 22 to 24 executing cases in one day, and the switch was left ON as the steady state.

What the landed tests check: (1) Clara dodged a direct yes/no question about weekend tour availability instead of answering it; (2) Clara dismissed a tour-confirmation card as "just a copy" instead of treating it as real. Both now regress automatically if they ever come back.

It also proved the safety story: reviews caught three real pipeline defects on the way — a test that would never actually run (file-naming mismatch), a required hand-typed quote that contradicted the "👎 is the only human step" rule, and an email-convention clash between the two repos. Each time the pipeline paused itself, reported honestly, and shipped nothing broken.

Remaining wiring for fully hands-off nights: the scheduled job's environment still needs the test-library and Slack credentials (today's supervised runs used local ones; one Slack note was posted by hand when the local credential was missing).

3 · Tests guard every change — the gate PROVEN BY STING

We deliberately tried to sneak five bad changes and two good changes past the gate in draft PRs. Every in-scope bad change went red within about a minute: a weakened check, a deleted test, a faked answer key, a hand-patched engine. Both good changes stayed green — no crying wolf. The leasing gate now executes the pipeline's own landed cases (count verified going up as they land).

Honest limit, proven concretely: the gate replays saved conversations; it never runs live product code. We put an incident-shaped mistake straight into Clara's reply code and the gate didn't blink. The gate stops known regressions from returning — it is not a general net for new behavior changes. New mistakes enter the net the moment someone grades them (stage 2).

4 · A judge grades live conversations LIVE, CALIBRATED

The placeholder that missed every mistake Fede caught is replaced by a real AI judge, calibrated the honest way: tuned on 70% of his 179 verdicts, scored once on the untouched 30% — 87% agreement, 86% of real problems caught. Only the two problem classes that proved 90%+ trustworthy may page #alerts (unbacked "you're all set" claims, ignored escalation requests); everything else writes quiet grades. Runs on the cheapest model tier (a few cents a day); a judge outage falls back to the old detector instead of stopping grading.

5 · Operators see what Clara did LIVE ON PROD

Every Clara reply now carries a small always-visible verdict chip ("● Delivered", "● Guard caught it") right under the bubble, in the panel PMs actually use; clicking it opens the factory view (message in → context → reasoning → tools → outcome door) for that exact turn. Placement chosen by a 5-designer, 3-judge UX panel after the first placement failed discoverability. Guard-blocked turns get their own chip (shipped as a same-day gap fix).