Superseded as the living doc → the Eval & Testing Roadmap (this page stays as the frozen record, 2026-08-20).
COMPLETE — every number independently re-runCatching Clara's Mistakes
The three-layer testing proof — built, measured, stress-tested
2026-08-20 · Four competing prototypes vs. five real Camellia conversations that actually failed. Companion to the Quality System Plan.
Verdict: compose, don't adopt. Our own TypeScript checkers + promptfoo as the runner + one shared before/after diff + the AI judge only for wording-level faults. All four lanes caught 5/5 real failures, 0 false alarms, 2/2 seeded regressions — and all of it reproduced when independently re-run. The audit's real finding: structure-based checks generalize; phrase-based checks memorized the answer key.
5/5 ×4real failures caught, correct layer + verdict, every lane — independently reproduced
0 ×4false alarms on clean real conversations
2/2 ×4seeded regressions killed — the "did a change make it worse" primitive works
$0metered spend across all builds and re-runs
6.7%how much of the best framework lane's code the framework actually contributed
B B C Boverfit grades — the reason perfect scores aren't the headline
How a turn gets checked
Every Clara turn produces three records
Context
what Clara was told
what Clara was told
Truth
what was actually true — serialized text, no live database
what was actually true — serialized text, no live database
Trace
what tools ran & returned
what tools ran & returned
Draft
what she's about to send
what she's about to send
↓
Three gates, cheapest first
L1 · Input
context vs truth diff · thread-ownership rule
context vs truth diff · thread-ownership rule
→
L2 · Loop
draft claims vs actual tool results
draft claims vs actual tool results
→
L3 · Outbound
numbers vs truth · promises backed · narration leak
numbers vs truth · promises backed · narration leak
↓
pass
send as-is
send as-is
repair
fix the exact span, cite the truth row
fix the exact span, cite the truth row
hold
human takes the next turn
human takes the next turn
block
kill it, page engineering
kill it, page engineering
Verdicts always cite their evidence — a field diff, a tool result, a span. A judge reading one transcript cold is banned (measured ~15–25% accuracy on "which layer broke").
The five real failures it had to catch
| # | Layer | What really happened (sanitized, verbatim otherwise) | Verdict |
|---|---|---|---|
| 1 | INPUT | Current tenant told "you moved out" — stale residency record fed the context | block |
| 2 | INPUT | Auto-replied on a thread a human teammate already owned | hold |
| 3 | LOOP | Tool failed; Clara presented the failure as policy ("that's just how the process works") | repair |
| 4 | OUTBOUND | Internal system narration sent to a prospect — the "kill it, page engineering" case | block |
| 5 | OUTBOUND | "The team will follow up" with nothing behind it — the class prod now catches, so it doubles as a known-fixed-bug test | repair |
Plus two clean real turns as false-alarm bait, and a regression set: fixed versions of the 5 failures + deliberately worsened versions of the 2 clean cases.
The four lanes, verified
| Lane | Wall time | Code | Overfit | Keep? | One-line reason |
|---|---|---|---|---|---|
| Hand-rolled TypeScript | ~1.3s | 1,313 lines, zero deps | B | KEEP — the checkers | Best evidence-cited verdicts; nothing to depend on or be abandoned |
| promptfoo-extended | ~42s | ~850 lines glue | C | KEEP — the runner | Checks drop into the format of our 56 existing suites; login just works; CI reporting already wired |
| Guardrails AI | ~0.05s | framework wrote 6.7% | B | PASS | Proved the serialized-truth thesis, then contributed scaffolding + one silent bug (chained validators drop all but the last, no warning — reproduced) |
| inspect-ai (Python) | ~1.0s | ~1,026 lines | B | PASS | Its one differentiator (model-graded scoring) went untested; a Python sidecar isn't earned |
All four reproduced on independent re-run. Discrepancies found and disclosed: one stale line-count (3% under), one wall-time variance band, one run-to-run drift in the unscored judge leg.
The finding that matters
Structural checks — wording-robust, schema-brittle
truth-vs-context diffs · dollar figures vs ledger · tool succeeded · thread ownership. Zero wording dependence — survived every paraphrase and 100% of 560 perturbed cases. But schema drift (renamed/nested fields, numbers-as-strings) broke 17/20 attacks and crashed 2 — production-hardening + unit tests required before the grade-A label is earned.
truth-vs-context diffs · dollar figures vs ledger · tool succeeded · thread ownership. Zero wording dependence — survived every paraphrase and 100% of 560 perturbed cases. But schema drift (renamed/nested fields, numbers-as-strings) broke 17/20 attacks and crashed 2 — production-hardening + unit tests required before the grade-A label is earned.
Phrase checks — confirmed memorizers
narration leak · invented policy · unbacked promise. Keyword lists copied from the answer key in all four lanes — and the bigger run measured it: 1/80 reworded attacks caught (1.3%), 17% recall on real over-promises, 7% on real hallucinations. These classes belong to the judge.
narration leak · invented policy · unbacked promise. Keyword lists copied from the answer key in all four lanes — and the bigger run measured it: 1/80 reworded attacks caught (1.3%), 17% recall on real over-promises, 7% on real hallucinations. These classes belong to the judge.
↓
Consequence — the wording-level third of the problem belongs to the AI judge, not more regexes. Where the judge did run, it recovered exactly what rules couldn't (hold vs block on a bad reply). Its full quality is the one unmeasured number — being measured right now (below).
How CI tells a PR it made Clara worse
PR opens
→
Old code
replies to the corpus
replies to the corpus
→
Checkers
verdict per case
verdict per case
→
Diff
which cases flipped
which cases flipped
New code
replies to the same corpus
replies to the same corpus
→
Checkers
same rules, same judge
same rules, same judge
↗
↓
Per-case flips, evidence quoted
"case 12: pass → block — draft now contradicts the lease status"
"case 12: pass → block — draft now contradicts the lease status"
→
Sticky PR comment
the same loop the review bot already reads — the builder fixes it before merge
the same loop the review bot already reads — the builder fixes it before merge
Proven in the POC: every lane reported all 7 before/after flips correctly. Deterministic layer costs seconds per hundreds of cases; judge calls are serialized (concurrent use rate-limits the subscription — measured today).
How it connects to the golden set & the grading UI
Quality Desk
the annotations UI — founder & team grade real conversations
the annotations UI — founder & team grade real conversations
→
Golden set
385 graded rows and growing — the labeled corpus
385 graded rows and growing — the labeled corpus
→
Quality engine
replays the corpus with checkers + judge on every change
replays the corpus with checkers + judge on every change
↓
Disagreements
machine vs human verdicts land back in the Desk as review items — annotating them recalibrates the judge
machine vs human verdicts land back in the Desk as review items — annotating them recalibrates the judge
→
New paired cases
every prod incident + every overturned verdict becomes a permanent before/after test
every prod incident + every overturned verdict becomes a permanent before/after test
→
CI corpus grows
the bug that happened once is checked forever
the bug that happened once is checked forever
Same loop Sierra and Intercom run: humans annotate, the harness replays, disagreement drives the next annotation. The Desk already exists; the golden set already exports; the engine closes the circle.
The bigger run — measured
1.3%of 80 reworded attacks caught by the phrase detectors — memorization confirmed with a number
17% / 7%recall on the golden set's real over-promises / hallucinations (answer-key-blind, never-seen rows)
0false alarms on every testable founder-graded good reply
560 in 8mscases through the checkers at scale — 100% verdict accuracy, ~0.014ms per case
105 in 5mspaired before/after diffs — 30/30 regressions killed, 75/75 fixes validated
3/20structural rules surviving schema-drift attacks — the new hardening work-list
| Proof | What ran | Result | What it means |
|---|---|---|---|
| Generalization measured | Shipped detectors vs. all 385 founder-graded golden rows they had never seen | Over-promise recall 4/23 (17%) · hallucination 4/60 (7%) · good replies 0 false alarms · hand-audit: several "catches" matched the analyst's incident notes, not Clara's own words | Phrase rules don't generalize — as predicted. Two data findings: golden rows mix Clara text with analyst notes (checkers must only ever see Clara's words), and 24% of rows carry no reply text at all. The dominant real hallucination shape (fabricated facts, IDs, "already done" claims) is structural — it needs the truth ledger, exactly what the architecture builds. |
| Adversarial measured | 130 synthetic attacks: 80 reworded faults, 20 near-miss clean baits, 20 schema-drift attacks, 10 quoted-customer confusions | Paraphrases: 1/80 caught (narration 0/20, policy 1/20, promise 0/20, residency 0/20) · bait: 2/20 false alarms, both auto-block · schema drift: 3/20 survived, 2 crashes · quoting confusion: 7/10 | Three concrete bugs for the engine: nesting one field deeper silently defeats the human-ownership hold; the narration rule blocks on ordinary phrases like "on my end"; no detector separates Clara's voice from quoted customer text. The 130-item set is now a permanent fixture. |
| Scale measured | 560 perturbed cases + 105 before/after pairs through the unmodified checkers, reproduced 5× | 7.9ms total (0.014ms/case, p95 0.018ms) · 560/560 correct verdicts · pairs: 5.4ms, 100% kill + fix-validation | The deterministic layer is free at any corpus size — running it on every PR, on the full corpus, costs nothing. CI cost lives entirely in the judge leg. |
| Judge vs. founder measured | 147 golden rows judged label-blind by claude-sonnet-5 (the planned judge model), exact rubric, one verdict per row | Agreement 69.9% (72/103 rows with text) · miss rate 14.4% (judge said clean on 14/97 founder-flagged rows) · false alarms 2 of 6 testable good replies · top confusions: over-promise read as wrong-fact (6), hallucination read as clean (6) | This is the floor: a context-free judge reading excerpts alone. It already beats the phrase rules by ~4–10× on recall, but 2-of-6 false alarms (tiny sample, but directionally over the 2% trust budget) says an excerpt-only judge must never block on its own — it needs the receipts + truth the checkpoint feeds it, and clean-reply calibration on a much bigger sample. Method note: judged in agent batches (the direct API path was device-throttled — account rotation disproved quota as the cause); disclosed in RESULTS.json. Adjudication of all 31 disagreements against the founder's verbatim verdicts: only 1 was a mislabeled row (the machine-inferred-label worry proved small) · 6 confirmed judge errors · 11 genuine taxonomy overlaps (an unbacked "done!" claim is both an over-promise and a false statement — the classes need a precedence rule) · 13 were unfair tests (the excerpt never contained the evidence — the context-free ceiling, not judge failure). Adjusted agreement 70.9%. Deeper data finding: 27 of the 31 rows carry no raw founder quote at all, only a reconstructed summary — golden-set provenance needs rebuilding, not just relabeling. |
What gets us to grade A
- Make more checks structural. Grade-A rules exist wherever truth is captured — the receipts + serialized-truth work in the Quality System Plan converts fault classes from "wording" to "field diff."
- Judge owns the semantic classes, with its agreement against founder grades measured (in flight above) and phrase lists demoted to cheap pre-filters.
- Build blind, test reworded. Next fixture round: one team writes cases, another writes checkers; every class ships with paraphrase variants (the adversarial set above is round one).
- Unit-test the detectors and harden the schema seams — the adversarial run crashed 2 structural rules and defeated the ownership hold by nesting one field deeper; tolerant accessors + per-detector tests close it.
- Separate Clara's voice from quoted text — 7/10 quoting confusions say detectors must run on Clara's own words only (same lesson as the golden set's analyst-notes contamination).
Honest edges that remain
- The judge numbers are a context-free floor — the real checkpoint feeds the judge receipts, truth, and the trace; measuring that lift is the next run.
- Clean-reply calibration is under-sampled: only 6 founder-graded good replies had excerpt text (44 of 147 sampled rows had none — a golden-set data-quality item in its own right). The false-alarm rate needs hundreds of clean rows before it's a rate, not a signal.
- Judge cost at corpus scale is still projected, not measured — the direct-API throttle prevented per-call latency numbers today.
Method: 4 independent builder teams, one sealed fixture pack (sanitized real conversations — zero real names, proven by programmatic scan), one scorecard frozen before any build, hard rules (subscription auth only, offline replay only, nothing merged). Then 4 adversarial verifiers re-ran every harness from scratch and audited every checker for answer-key overfitting. Full per-lane write-ups, code, and the fixture pack move into the quality-engine repo once it's named.