The Eval & Testing Roadmap
Every known gap in how we test Clara, what fills it, and when — the living tracker. Measured evidence lives in the archive pages below.
Verdict & the numbers
No AI-behavior check is a required merge-blocker yet — this page tracks every gap until that changes. The eval inspection (2026-08-19) found the pattern; the Quality System Plan sequenced the fix; the three-layer POC (2026-08-20) proved the missing primitives actually work. This tracker holds 34 known gaps — the inspection's original 10, this week's measurements, and the 2026-08-22/23 Cerberus/Quality Desk update — each with a status and an owner until it closes. Update 2026-08-22/23: the leasing regression gate now runs (advisory) on every PR, the Cerberus engine repo migration closed, and the Quality Desk's own reload bug closed — but a real incident (case-22, row 32) confirmed the required gate's own domain rule would have routed it to advisory, not required, and the voice channel remains entirely outside this gate's reach (row 33).
0AI-behavior checks in branch protection today — the leasing gate runs on every PR but isn't a required check yet (deliberate burn-in)
1 in ~200umbrella-sweep attempts that actually execute — a daily cost cap skips the rest, and a skip still renders green
1.3%of 80 reworded attacks the phrase-matching detectors caught — memorization, confirmed with a number
70.9%judge-vs-founder agreement reading an excerpt cold, no receipts, no trace — adjusted after full adjudication (was 69.9%); the floor this roadmap has to beat
20/20reproduction rate for the 2026-08-22 "you're all set" unbacked completion claim, offline, with the current tool wording
34known gaps tracked below, each with a fill, a phase, and a status
The three-layer model
Every check on this roadmap slots into one of three gates, cheapest first — the architecture the POC proved out on 5 real failures with 0 false alarms.
Three gates, cheapest first
L1 · Input
context vs truth diff · thread-ownership rule
context vs truth diff · thread-ownership rule
→
L2 · Loop
draft claims vs actual tool results
draft claims vs actual tool results
→
L3 · Outbound
numbers vs truth · promises backed · narration leak
numbers vs truth · promises backed · narration leak
↓
pass
send as-is
send as-is
repair
fix the exact span, cite the truth row
fix the exact span, cite the truth row
hold
human takes the next turn
human takes the next turn
block
kill it, page engineering
kill it, page engineering
Verdicts always cite their evidence — a field diff, a tool result, a span. A judge reading one transcript cold is banned as the sole verdict source (measured ~15–25% accuracy on "which layer broke").
The gap board
Plain-English gap · where the evidence comes from · what closes it · which phase it's scheduled in · current status. Every row traces to the archive pages linked at the bottom.
| # | Gap | Evidence | What fills it | Phase | Status |
|---|---|---|---|---|---|
| 1 | Nothing can block a bad AI-behavior change from merging — only ordinary code checks (build, types, unit tests) stop a merge today. | inspection | EXPANDED (2026-08-22/23): "Regression Gate — Leasing (required)" now runs on every PR, not just the proof folder, plus an advisory gate and an advisory judge lane (claude-sonnet-4-6, subscription-billed) — #6009 #6045. Still not in branch protection — deliberate burn-in period, not an oversight. Case-22 (row 32 below) shows a real incident the leasing lane's own domain rule would route to the advisory lane, not the required one — evidence for the eventual expand decision, not against it. | Wk 2–3 | IN PROGRESS |
| 2 | Voice's two purpose-built safety nets can't report a real failure — one is switched off, the other is permanently "soft" and can only ever say success. | inspection | Two small, already-diagnosed mechanical fixes: turn the nightly sim back on; remove the soft flag. | Next | NOT STARTED |
| 3 | Any future model comparison inherits two live bugs — a determinism setting that silently vanishes for newer models, and a reply budget too small for models that reason before answering. | inspection | DONE (2026-08-21): the harness shipped and the first fair comparison ran — all 8 lanes, 332 cases, 5 draws per case per model, blinded pinned judge, honest recording, zero errored rows, all integrity canaries green. Verdict: NO SIGNIFICANT DIFFERENCE overall (pooled p=0.56 across the two lanes with enough disagreements to measure; five lanes had zero disagreements at all; the renewal lane leaned toward today's model on all 3 of its disagreements but is too small to call). Case-level flips exist in both directions and are listed in the run report — including one fair-housing escalation case worth an eyeball. The permanent asymmetry (newer models reject the determinism setting) is disclosed in every report by design. Any future model question is now a button press. | Done | DONE |
| 4 | The pipeline that turns a human thumbs-down into a permanent, automatic fix is fully built — four green, reviewed PRs sitting untouched. | inspection · plan | Meet the numeric unlock bar (Decision q-unlock-bar below) before merging — never re-raise the merge question itself. | Held | DECIDED-HELD |
| 5 | The two highest-stakes decisions in the system — was the maintenance call routed/priced right, was the renewal decision right — have a grading slot and nothing feeding it. | inspection | Build the two judges; the plumbing for one already sits in another held PR. | Later | NOT STARTED |
| 6 | A handful of suites run in zero automated places, and ~8 more domains have no nightly safety net — only whichever change happens to trigger the once-daily umbrella first. | inspection · plan | Nightly full sweep retires the cost-capped umbrella; per-property checks; folds into the engine migration. | Wk 3+ | PLANNED |
| 7 | A real fix to how Clara flags renewal counter-offers shipped without its test schema updating — tests can only check that some escalation fired, not the right one. | inspection | Add the missing field to the renewal test schema — small, targeted. | Next | NOT STARTED |
| 8 | The plumbing behind "is this actually passing" is fragile — a bug that could hide a real red result behind a stale green was already found and fixed once. | inspection | No dedicated fix scheduled yet; the paired before/after diff (Wk 2–3) reduces exposure by comparing real runs instead of trusting one green. | Later | NOT STARTED |
| 9 | 35 test files each hardcode their own copy of the judge model, with nothing checking they match the officially pinned one. | inspection | One shared judge library in the engine repo — every suite that adopts it retires a private copy. | Next | PLANNED |
| 10 | One check passes at a 60% bar on one system and only a 95% bar on another — same code, same day, a 35-point gap the code's own comments call the worst conflict in the repo. | inspection | Align the two checkers' pass bars — contained fix. | Next | NOT STARTED |
| 11 | The phrase-matching detectors don't generalize — they memorized the answer key: 1.3% caught on 80 reworded attacks. | POC / bigger run | Judge owns the semantic classes; phrase lists demoted to cheap pre-filters only. | Wk 4 | IN PROGRESS |
| 12 | The structural (schema-diff) checks are wording-robust but schema-brittle — drift attacks defeated 17 of 20, crashed 2, and beat the human-ownership hold by nesting one field deeper. | POC | CLOSED in the POC engine (2026-08-21): the ownership check now finds its flag at any nesting shape via a bounded deep search instead of a fixed path list, and null/missing data returns a structured "cannot verify" instead of crashing. Every fix is pinned by the verbatim attack as a permanent test case — including one nested a level deeper than the original attack. Honest limit stated in the repo: the search stops at depth 4. | Wk 1–2 | IN PROGRESS |
| 13 | No detector separates Clara's own words from quoted customer text — 7 of 10 quoting-confusion attacks fooled the checkers. | POC | CLOSED in the POC engine (2026-08-21): a small grammar now separates Clara's own words from customer-quoted text; all 10 quoting-confusion attacks pass (was 3 of 10 fooling the checkers after the first fix attempt — the review bot caught the first grammar over-masking, and that got fixed too). The 3 attack payloads are permanent test cases. | Wk 1–2 | IN PROGRESS |
| 14 | The AI judge, reading an excerpt cold with no receipts or trace, agrees with the founder only 70.9% of the time (adjusted after full 31-row adjudication against his verbatim verdicts; was 69.9%), misses 14.4% of real problems, and false-alarmed on 2 of 6 testable clean replies — over budget, on a tiny sample. The adjudication breaks down as 1 label-noise, 6 judge-wrong, 11 both-defensible (taxonomy overlap — an unbacked completion claim is legitimately both an over-promise and a false statement), 13 insufficient (the excerpt lacked the evidence to call it — the context-free ceiling, not a judge error). | POC / bigger run | Feed the judge the real checkpoint context (receipts + trace, not an excerpt); build a much bigger clean-reply calibration set; add a verdict-class precedence rule so overlapping fault classes (e.g. over-promise vs. false-statement) resolve to one call instead of splitting the disagreement count. | Wk 5 | PLANNED |
| 15 | The golden set has real data-quality holes — 44 of 147 sampled rows carry no reply text at all, some rows mix Clara's words with an analyst's incident notes, and 27 of the 31 judge/founder disagreement rows turned out to have no raw founder quote at all, only a reconstructed summary. | POC · plan | Schema cleanup + verbatim-first labeling discipline — for the 27 unsourced rows specifically, a provenance rebuild back to the founder's actual words, not just relabeling. | Wk 1 | IN PROGRESS |
| 16 | No answer-key-blind fixture discipline exists yet — the same pass that writes a case also writes the checker for it. | POC | Blind authorship + paraphrase variants per fault class; the 130-item adversarial set built for the POC is round one. | Wk 1 | IN PROGRESS |
| 17 | A grading-bench bug: the founder's own saved grades appear to disappear on reload — and the golden set sits downstream of this bench, so the bug is upstream of everything else on this page. | today's incident | CLOSED (2026-08-22): from the founder's first real annotation session on the renamed Quality Desk — pinpoint saves fixed, handles multi-message turns, reads back the prior verdict instead of looking blank. First-session annotations verified landing in the GradeVerdict corpus. (#6110) | Wk 1 | DONE |
| 18 | Eval infrastructure has sprawled — an 18MB untracked script tree, ~15 scripts named after real people, 30+ copy-pasted judge implementations. | inspection · census | DONE (2026-08-22): burn-in evidence pack (the burn-in report) cleared the switch-over. The engine — Cerberus v0.1.2 — is consolidated into its own standalone repo (PropFlow-Technologies/cerebrus) and vendored into quality-gate-poc/, tag+checksum pinned, synced via scripts/sync-cerebrus.sh. Corpus is 21 golden fixtures. One gap remains: case-22 (row 32 below) is upstream in cerebrus, not yet in the vendored corpus. (#6039 #6072) | Wk 1–2 | DONE |
| 19 | Judge calls have to run one at a time — a device-level API throttle was measured today, and it directly caps how fast any CI job calling the judge can run. | plan | A pooled/serialized judge client, built into the engine. | Wk 2 | NOT STARTED |
| 20 | The metered-spend guard overreaches — it pattern-matches file contents rather than actual spend risk, costing real engineering time on false positives. | today's incident | A scoped tune-up to the guard's matching logic. | Later | NOT STARTED |
| 21 | Conversational email cannot be exercised end-to-end by any test lane — the designated test mailbox is hard-wired to automated data reports and silently skips prospect-style emails (predates the reason-first change; two identical day-before failures on record). Any email-drafting change ships provable only by suite evals, never by a live round-trip. | post-deploy | Stand up a test mailbox (or route) that reaches the conversational drafting path, then add the email round-trip to the post-deploy runbook next to the SMS smoke. | Next | NOT STARTED |
| 22 | No eval lane speaks Spanish — a Spanish-language prospect got invented pet fees and made-up parking prices, and not one of our suites would have caught it (all cases are English). | fleet 8/21 | Add Spanish variants of the fee/price and escalate-the-unknown cases to the leasing and renewals lanes, plus a grounding rule: a fee or price answer must match the knowledge-base row exactly or escalate — never estimate. | Next | NOT STARTED |
| 23 | No life-safety test lane exists that runs regardless of who the sender is — a stranger texting "I smell gas" got a brush-off, because the safety reply only fires for known tenants and no suite covers the unknown-sender path on any channel. | fleet 8/21 | An always-run life-safety suite across every channel and sender state (known, unknown, unverified), shipped alongside the gas-leak fix itself. Highest-stakes row on this board. | Wk 1–2 | NOT STARTED |
| 24 | Spoken promises are never checked — Clara told a caller "the leasing team will reach out" and nothing was logged, nothing noticed. The promise-backing check that already works on text has no voice equivalent. | fleet 8/21 | A post-call sweep (never inline — voice latency rule) that reads call transcripts for commitments and verifies each one created its backing record. | Next | NOT STARTED |
| 25 | The adversarial test bench silently under-covers — the test property's tours-only setting parks scenarios before Clara's judgment ever runs, and the test-sender address style trips the spam filter, so a fair-housing scenario was skipped WITHOUT saying so. A silent skip reads as coverage. | fleet 8/21 | A dedicated verified test sender, a bench exemption for addressed-to-Clara mail, and the no-silent-skips rule: every skipped scenario must appear in the run ledger by name. | Next | NOT STARTED |
| 26 | Nothing detects one property's data inside another property's knowledge base — the Willows lease-terms entry literally says "Camellia Apartments offers…", and no check would catch Clara reading it to a Willows prospect. | fleet 8/21 | A knowledge-base contamination scan (each property's KB checked against every other property's name and identifiers) + a one-off scrub of the cloned entry. | Next | NOT STARTED |
| 27 | Golden data sets have only one intake — desk grades. The team's live review sessions carry the same judgments ("that reply was wrong, here's why") and none of it is captured. | Fede 8/21 | Automatic intake from the Slack transcripts channel: read the raw per-speaker transcript of each review session (never the recap), extract the concrete rulings, and propose each as a golden-set entry / corpus case for human confirmation. Queued behind the leasing replay spike. | Next | PLANNED |
| 28 | The checker misses a contradiction inside a single reply: in a replay, Clara said "tour confirmed" one line after the booking call was denied — and the checks still passed it. For leasing there is no independent "what was actually true" record wired in yet, so the truth-comparison layer runs blind there. | replay 8/21 | Wire a truth snapshot for leasing replays (the recorded tool denials ARE the truth source) and add a same-turn contradiction rule; the failing example ships as a permanent test case. | Wk 2 | NOT STARTED |
| 29 | The burn-in proved one whole check is unprotected: the "did she actually do what she claims" detector can be silently switched off with zero visible change — no test case in the corpus exercises it in either direction. And an already-disclosed known-miss can be quietly rebranded as "fixed" with no signal. | burn-in 8/21 | Add fixtures that exercise every detector in both directions (a detector-coverage matrix asserting each check has at least one catch and one clean case), and require known-miss status changes to cite the engine change that earned them. | Wk 2 | NOT STARTED |
| 30 | Nobody watches the watcher: a single commit can neuter the regression-diff tool and forge its own integrity checksum at once, and a hand-written "all passed" results file is accepted without proof a real run produced it — every automated layer defeated together. | burn-in 8/21 | Provenance, not self-consistency: CI recomputes the vendored checksum against the real Cerberus repo (needs the org's first cross-repo CI credential), verdict files get a run-attestation, and changes to the diff tool itself always page a human. | Next | NOT STARTED |
| 31 | Post-hoc prod conversation judging is dark since ~Aug 16 — the daily reviewer (conversation-review-daily) is paused, the planned cutover to sole reviewer never happened, and the replacement being planned (a real-time #apartment-camellia activity feed) is an activity feed, not a judge. | verified 8/21–22 | See the new "Post-hoc prod judging" section on the Quality System Plan and its decision, q-posthoc-next. | Proposed | NOT STARTED |
| 32 | Real 2026-08-22 incident: Clara told a prospect "you're all set" on a tour reschedule that never actually happened. Offline eval reproduces it deterministically — 20/20 with the current tool wording, 0/20 on both honest wordings tried. The case is upstream in cerebrus (case-22, not yet in the vendored corpus) and classifies to "other" domain by the mechanism rule — meaning the REQUIRED Leasing gate (row 1) would not have caught it; only the advisory lane would. | incident 8/22 | Land case-22 upstream-first: cerebrus PR #2 → founder merge + v0.1.3 tag → sync PR #6108 into quality-gate-poc/. Separately informs the eventual required-vs-advisory scope call for gate row 1. | Wk 1–2 | IN PROGRESS |
| 33 | The voice channel sits entirely outside the outbound safety pipeline — it runs inside ElevenLabs' own conversation loop and never touches the SMS/email safety checks that gap 32's fixture exercises. No amount of hardening the text-side gate reaches a voice call. | 6112, 8/22 | Founder-commissioned 2026-08-22: prior-art + ElevenLabs best-practices research, an options catalog, a throwaway-agent latency prototype, then a decision doc. Constraint: no new hot-path guard without latency evidence; preferred shape is a tool-level check-before-confirm with conflict-explicit tool feedback. Nothing voice-related merges without explicit founder approval. | Next | NOT STARTED |
| 34 | The Quality Desk queue split one person's conversation across channels, and its ranking scored a placeholder rubric instead of real risk — annotators were working a fake priority order. | annotation session 8/22 | CLOSED for ranking (2026-08-22): retired the ranked inbox for one newest-first, person-legible list (ADR-0128 Amendment 2). Grouping the list by person across channels is in flight. (#6106) | Wk 1 | IN PROGRESS |
The roadmap
Same shape as the Quality System Plan's five-week sequencing, extended with every gap that plan didn't already schedule. Founder ruling stands throughout: decisions wait for working software, nothing arms, and the four held PRs (gap 4) stay held.
| Phase | Theme | Ships | Gaps |
|---|---|---|---|
| Wk 1 now – Aug 24 | One ruler | Case format + LAYER tags; corpus fold-in (golden 385 + would-flag bed + top suites); baseline scoreboard; golden-set schema cleanup + verbatim-first labeling; blind-fixture discipline; grading-bench persistence bug closed; Quality Desk queue de-ranked to a single person-legible list. | 15, 16, 17 (done), 34 |
| Wk 2 Aug 24–31 | CI delta — prove the mechanism | Paired base-vs-head spike on the intent lane; context receipt on AgentTrace; pooled/serialized judge client designed against today's measured throttle. | 1, 19 |
| Wk 3 Aug 31–Sep 7 | CI delta rollout + input truth | Paired job extends to renewals; nightly main sweep retires the cost-capped umbrella; nightly truth-diff job; suites running nowhere folded in or retired. | 6 |
| Wk 4 Sep 7–14 | Ledger + checkpoint + hold queue | Truth ledger v1; checkpoint v2 widens to the full trace; the judge takes over the semantic classes the phrase detectors were memorizing. | 11 |
| Wk 5 Sep 14–21 | Prove offline, drill the hold | Offline subscription replay of checkpoint v2 over real traffic, precision/recall published — the number that answers the judge-floor question with full context instead of a cold excerpt. | 14 |
| Held | Per founder decision | The four thumbs-down-to-fix PRs stay exactly as decided — untouched until the numeric bar (q-unlock-bar) is met. Never re-raised outside that bar. | 4 |
| Next Wk 6–10 | Everything scheduled but not yet started | Voice's two safety nets; the voice hallucination-guard decision doc; the two model-comparison runner bugs; the renewal test-schema field; the 60-vs-95 threshold conflict; POC hardening (schema-brittle detectors, Clara-voice-vs-quote attribution); the shared judge library. | 2, 3, 7, 9, 10, 12, 13, 33 |
| Later Unscheduled | Real, but no committed date | The two empty high-stakes judge slots; the stale-green plumbing fragility; the spend-guard tune-up. | 5, 8, 20 |
Decisions
Founder ruling, 2026-08-20: "I need working software before decisions." Nothing on this roadmap blocks build work, and none of the six questions below is being asked right now. Weeks 1–3 and the POC proceed on the do-nothing-live defaults: everything stays offline, nothing arms, the four held PRs stay held. Each question comes back to the founders only once the working software that informs it exists — the POC results for the framework-shaped calls, the paired-CI spike numbers for the gating calls, the offline replay numbers for the checkpoint calls. They're pre-registered here so the eventual answers stay honest.
Answers save for everyone who opens this page; the recommended option is listed first and marked.
q-gate-lanes — Which suites become blocking first?
Blocking CI gates on AI behavior are already authorized in principle (D3, 2026-08-18) — this is about which checks go first. The same open call sits on the eval inspection page (frozen record); deciding it here supersedes that instance with the full five-week sequencing attached.
q-hold-host — Where do held messages live?
The fleet's ranked recommendation: extend the live /review queue rather than build a new surface — it already has the approve/reject mechanics, four prior extensions of the exact same pattern, and live polling; it just needs new platform_admin-only gating.
q-checkpoint-next — After the offline replay proves precision, what is the first live step?
The pre-send gate stays OFF today per the founder's standing decision. This decision only pre-registers what evidence would justify the next step — nothing moves without his explicit yes, even on green replay numbers.
q-unlock-bar — The numeric bar that unlocks the four held promotion-pipeline PRs
Fede's decision on 2026-08-20: the four PRs (#5782, #5784, #5785, #5786 — distiller, corpus re-run gate, promotion arm, principle injection) stay held "until agentic quality and testing significantly improve." This turns that sentence into numbers.
q-repair-scope — What may the checkpoint repair automatically?
The live "make it true" pattern today only ever repairs a claim class where the repair tool's success and the claim's truth are the same event (an escalation claim, backed by actually calling the forwarding tool) — never an arbitrary fact.
q-input-fix-priority — Biggest input-layer fix to schedule right after the five weeks?
Three real candidates surfaced tonight, each already documented as a known, named gap in the code.
Archive
The measured evidence and full method behind every row above.
- The Quality System Plan — the plan record: sequencing, scope fences, the six decisions above in their original context. 2026-08-20.
- Catching Clara's Mistakes — the measured evidence: four prototypes, five real failures, the numbers behind gaps 11–16. 2026-08-20.
- The eval system, deep inspection — five rebuilds since March, the Sonnet 5 story, the original ten gaps (1–10). 2026-08-19.
- The eval inventory — the census: every one of the ten testing mechanisms actually running today. 2026-08-20.
- Cerberus Burn-In — the evidence pack that cleared the Cerberus engine-repo migration (gap 18): 53 hand-built trials, 9 real findings, all in the safety net around the checker rather than its detection itself. 2026-08-21.