Inventory · verified from source, git history, and workflow files · 2026-08-20

Every way we test Clara today, in one list

Commissioned by Fede's question: "how can we consolidate all of our eval solutions into a single standard for me and Gera?" This page is the factual ground: ten distinct mechanisms, what each covers, and where they overlap or leave holes. The consolidation decisions belong in the Quality System Plan (in draft).

The ten mechanisms

#	Mechanism	What it is	How it runs
1	Promptfoo suites	56 configs in `evals/` (leasing, renewals, maintenance, fair housing, adversarial…), custom subscription-auth provider, docs hub at `docs/evals/README.md`	`bash evals/run-all.sh` manually; renewal + intent subsets on nightly/weekly crons; PR fast-lanes per domain
2	Subscription eval runners	~85 one-off TypeScript runners in TWO parallel trees: `scripts/eval-*-subscription.ts` (~34) and `model-evals-fleet/runners/` (~40, overlapping domains)	Manual scripts, OAuth bearer, never metered keys
3	Coworker harness	10 scenario files + LLM judge + adversarial seeds, live-loop against the Willows bench in real prod infrastructure	Manual only; hard-locked to the bench property
4	Mailbox invariants	Last-mile checks reading the real Graph mailbox (threading, duplicates, sender identity) — built after a bug the application-layer proof missed	Manual, inside the coworker harness; bench only
5	Drift tests / @sensor	275 `*.drift.test.ts` files, each pinning one past incident or invariant, machine-readable `@sensor` tags, registry printer script	Ordinary unit tests — run in the required CI check
6	Nightly crons	10 scheduled workflows: SMS/email stress, voice seam matrix, Camellia voice regression, renewal harness, suppression canary, ElevenLabs config + prompt drift, schedule-manifest drift	GitHub Actions cron, nightly/6-hourly/weekly
7	Voice harness	140 scenario YAMLs, 5 driver tiers (simulate → replay → robot calls → post-call E2E → call sequences), robo-vendor caller fleet, replay corpus	Two nightly crons + manual tiers for real calls
8	Five-incident replay	`adr-0054-incident-replay`: five real May–June incidents pinned red/green in a manifest; CI fails if a case has no test	In the required unit-test check, fully offline
9	LLM judges	30+ separate judge implementations, one per suite — no shared module exists; some suites keep an "-improved" judge copy alongside the original	Inside whichever suite owns each copy
10	Escalation CRUCIBLE	Graded ship-gate for the escalation system against a numbered R1–R130 requirements ledger; guard-railed live mode with kill file	Logic mode in CI; live mode manual, bench only

What's duplicated

JUDGES 30+ copy-pasted LLM judges, zero shared code. The fast-tier folder alone has ~15 near-identical copies with three different file-naming conventions.
RUNNERS Two whole trees of subscription runners covering the same domains (scripts/eval-* vs model-evals-fleet/) — and the second tree is not in git at all: no history, no review, invisible to every CI gate.
VOCAB The "no filler / no stalling" phrase lists are spelled out independently inside each voice-agent prompt — no shared constant. (This week's filler-reply incident makes this one concrete: detection and prevention must share one vocabulary or they drift.)
SCHEDULING The Monday renewal-eval cron was consolidated into the nightly run, but the old workflow file and its cron still exist "to drive the PR fast-lane" — the same scheduling logic in two places.

The holes

NOT ENFORCED The "prompt change must include an eval diff" check is not in branch protection — it warns, it doesn't block, and its escape hatch (no-eval-needed: in the PR body) needs no sign-off. It has already been widened twice reactively after prompt changes slipped past its filters.
ONE PROPERTY Every live harness (coworker, crucible, voice real-call tiers, vendor batteries) is hard-locked to the single Willows bench. Nothing live-tests any other property shape — which is exactly how this week's Camellia staff-reply incident sailed through: the harness couldn't look where the damage happened. (A nightly Camellia containment check is being added now.)
MANUAL-ONLY The coworker harness, crucible live mode, and two E2E workflows only run when a person remembers to run them. Named tests can silently mean their opposite (a scenario named "staff email gets no reply" had been flipped to assert staff DO get replies — and passed).
STILL RED The five-incident replay suite still carries known incident classes marked as reproducing (two at the verbatim-replay level); nothing outside that suite tracks the debt.
ONE OWNER Nearly every mechanism is single-author (Fede's automation account). Gera has one commit in crucible, zero in the coworker harness — today "the standard" lives in one person's head.

Where the decisions go

This page is deliberately facts-only. The consolidation choices — one judge library, one runner tree, one enforced prompt-eval gate, per-property harness coverage, what Gera owns — belong in the Quality System Plan doc currently in draft, which should treat this inventory as its ground truth.

Method: repo source + git history + live branch-protection API, gathered 2026-08-20. Companion incident context: the Camellia RCA.