Decisions consolidated into the Clara-as-a-Coworker tracker, Phase 4 (single list) — answers there count.
Every test harness, eval rig, and replay system in the PropFlowAI codebase, in one place — plus the design for one shared skeleton so building a harness stays the house craft instead of the house sprawl.
Verified 2026-08-26 · direct find/grep/git-log against ~/.claude/propflowai (main) plus 4 named worktrees · cross-checked against the eval-quality trio (eval-inventory-2026-08-20, the eval deep inspection, the Eval & Testing Roadmap) and simplification-plan-2026-08-26 §2 bucket 4 · promptfoo layer: a separate audit is running, see promptfoo-audit.md — not redone here
The short version: at least 45 separate places in the codebase test, replay, or stress Clara's behavior — and they run on 5 different "is this actually true" answers for the same question ("did this conversation go right?"). Nine of them independently reinvent "replay a conversation and grade it." One tree of ~1,850 files isn't even in version control. A second confirmation pass tonight found four separate real-PII fixtures with zero scrubbing (not one) and the branch-protection discrepancy above. Nothing here proposes new architecture — it proposes one shared skeleton (a fixture shape, a runner, a grader list, two scrubbers, one coverage ledger, one proof-of-work stamp) that every existing harness can plug into without a rewrite, plus a doctrine and a build check so the next harness lands inside the skeleton instead of beside it.
gh api branch protectionIn plain terms: Fede loves building harnesses — that's a strength, not the problem. The problem is every harness gets built from scratch: its own way of describing a test case, its own way of running it, its own way of scrubbing private info out of it, its own way of proving the result is real and not stale. This page lists everything that exists today, shows the 9 "replay and grade" systems side by side so it's obvious which parts of which one are worth keeping, and proposes the one shared skeleton — so next time someone builds a harness, they build the SCENARIOS, not the plumbing underneath them.
Source: three read-only inventories done this morning (main app repo, Clara Bench and robot repos, this Atlas checked against main). The intro numbers above and everything below this section are the August 26 count, kept as history.
Answer · What we have · Consolidate · Extend · Build order · Questions for Fede
The one harness is the Day 0 replay, grown from "hold or send" to the whole turn (Clara's words, tool calls, work orders, AppFolio writes) across text, email and voice with a Spanish slice, run on every core PR, with the Clara Bench judges and the Willows live run stacked on top.
Two kinds of change, two proofs. A code refactor must replay byte-identical (recorded model answers, no model call). A prompt or model change is expected to change words, so it is proven on the bench judges and runs in shadow first (Fede's Oct 25 merge-gate ruling).
| Family | Count | Alive? | Blocks a merge? | Note |
|---|---|---|---|---|
| Replay engines (Day 0, model-turn, live loop, old engines) | 6 | Day 0 alive; 2 dormant | no | Day 0: 1,317 real records, 5,515 cells, manual, one Mac |
| One-gate replays | 19 | one-off | no | each re-asks one gate on real data |
| Promptfoo configs | 104 | 83 wired; 21 wired nowhere | no | 42 only run in the weekly stress run |
| Drift tests | 710 | alive | yes (Unit Tests) | 541 read source text, not behavior |
| Mutation controls | 853 | alive | yes (Lint) | the one place "this guard goes red" is enforced |
| Clara Bench (repo + old A–G runners) | 2 lineages | alive, manual | only the set lock | frozen set v1; skill still points at the old runners |
| Shadow loops | 2 | alive | no | both run from unmerged local worktrees |
| Voice suites | ~10 tiers | mixed; 3 dead | no | 330 scripted scenarios run soft; 8 live agents grade nothing |
| Robot tests (AppFolio) | 11 | alive | yes (robot CI) | Camellia only; no Western Slope or Situs baselines |
| Willows drivers | 9 + ~10 | alive; ~10 uncommitted | no | 9 scripts in git; today's drivers live in a scratchpad |
| Stress runs (text, email, voice) | 6 | red | no | weekly run last green Jul 13 |
| One-off proofs (subscription evals, small scripts) | 114 + 52 | 48 made since Sep 27 | no | 3 of 114 in any workflow |
Each group folds into one place, and the old copies are deleted only after the proof in the last column passes.
| Built more than once | Folds into | Deleted | Proof it is safe |
|---|---|---|---|
| 4 conversation-replay engines | Day 0 full turn | 3 | Day 0 reproduces each engine's recorded verdicts on its own records |
| 19 one-gate replays | Day 0 caller cells | 19 | same verdict on the same records at the matching caller |
| 3 more copies of the hold decision (incl. the fast-passing required gate) | Day 0 | 3 | Day 0 check is green and required before the old gate leaves |
| 5 tour-time replays | Day 0 tour lane (tour-day fixtures) | 4 | same offered times per record |
| 7 Willows escalation scripts | willows-live scenarios | 6 | each scenario re-run once: same PASS/FAIL, record ids logged |
| 7 identity suites | identity-harness + Day 0 fixtures | 6 | union of their cases, same result |
| 2 bench runner lineages | clara-benchmark baseline runner | 1 | same scores on frozen set v1 (hash-locked) |
| 2 shadow loops | Opus shadow arm, merged | 1 | same texts; production replies match turn for turn |
| 4 ways to test a call | robot call (live) + scripted simulate | 2 | one recorded call, both judges agree |
| 2 human-rating paths | Slack buttons label record | 1 | 52 chat grades imported, counts match |
| 2 guest-card email replays | guest-card corpus | 1 | same grade per shared card |
| 7 scrubbers | one shared scrubber | 6 | identical output over every fixture today |
| Total | 53 | plus 30 dead suites (no run in 30 days, no owner) |
| Gap today | Rung it serves | Fix |
|---|---|---|
| Day 0 sees only hold or send | 3 real-book replay | cells carry reply text, tool calls, filed records |
| Not in CI; snapshot only on one Mac | 3 | private off-Mac copy, compare on every core PR |
| No fixtures for data changes (new module state, a company going live) | 3 | fixture overlay on the frozen snapshot |
| No voice in the replay | 3 | real call transcripts and after-call records as cells |
| No simulated prospect or tenant attack (staff only) | 2 adversarial | simulated users on the subscription, two skeptics |
| No required check runs a behavior test | merge gate | Day 0 compare becomes required |
| Weekly stress run red since Jul 13 | 5 Willows | fix or delete; nothing hides behind red |
| 215 of 261 repair texts unrunnable (before the message mirror) | bench | rebuild their history from AppFolio, read-only |
| Live calls ungraded on 8 voice agents | 7 supervised | criteria on every agent (Fede's call, see Q3) |
| Spanish is smoke only (27 of 421) | bench + 3 | full slice, judged in Spanish |
| No coverage ledger (incident → fixture → red → green) | all | one ledger, rendered on this page |
| No shared scrubber | all | one scrubber (see Consolidate) |
| Willows drivers uncommitted | 5 | one committed willows-live suite |
Cut by proof: each PR merges alone, dark, under 600 lines. Owner is the Clara lane (001) unless marked.
willows-live suite (text, email, call) with the scenario log and record ids. Proof: today's Willows scenarios re-run from the suite.npm run harness entry point.Fede's call: only the three questions below. Everything else is built and merged under the standing grant.
August inventory, historical. The table and sections below are the Aug 26 count, kept as history. The current plan is October 27 plan: one harness above.
Every harness/rig found tonight, one row each. Click a column header to sort; use the chips or the search box to filter. Evidence for every row: file count and last-commit date are from direct find/git log against the repo tonight; CI status is from a path-scoped grep against .github/workflows/ (not a basename substring match — several early basename hits turned out to be false positives and were corrected). Where a fact couldn't be confirmed tonight, the cell says so instead of guessing.
shown
| Name | Path | Proves | Layer | Fixture format | Runner | Grader(s) | Scrubber | Corpus | CI status | Incident origin | Overlap with |
|---|
Nine directories independently solve "load a fixture, run it through something, diff/grade the result." None has everything. This is what each has that the others lack.
| # | System | Input source | Replay mechanism | Grading | Isolation (can it write prod?) | Unique strength |
|---|
Not a rewrite. A thin, shared layer every existing harness maps onto through an adapter — the harness keeps its own scenario files, its own domain logic, its own name.
Built directly from three real formats already in the repo: promise-harness/cases.ts, after-call-harness/manifest.ts (tonight's newest harness — already has two of the three attestation mechanisms below), and Cerberus's desk-row fixture format (quality-gate-poc/scripts/convert-desk-row-to-fixture.ts, per BUILD-SPEC.md §5).
interface HarnessFixture {
id: string; // canonical id, e.g. "after-call-001"
suite: string; // owning harness, e.g. "after-call-harness", "promise-harness"
kind: "case" | "invariant"; // "case" = replay a conversation/call; "invariant" = standing
// architecture check (e.g. voice-harness's live-tool-budget)
layer: "prompt-message" | "conversation" | "outcome-after-call"
| "live-call" | "infra-ci-integrity";
subLayer?: string; // domain vocab kept AS-IS, not renamed:
// Cerberus "input"|"loop"|"outbound", voice-harness "L1".."L4"
provenance: {
source: "real-transcript" | "synthetic" | "adversarial" | "prod-replay";
sourceRef?: string; // conversationId / call id / shard:line — byte-exact pointer
incidentIds: string[]; // named incidents this fixture proves fixed
piiVerified: boolean; // required true before a real-transcript fixture can land
scrubMethod: "log-mask" | "full-replace" | "none-needed";
};
input: unknown; // opaque per-domain payload — untouched, whatever shape the
expected: unknown; // suite already uses for its case/verdict
confidence: "confirmed" | "unconfirmed";
taxonomyRefs?: string[]; // e.g. classRef[] (systematic-pattern.md classes A-M), ["R42"]
regressionStatus: "red-before-green-after" | "already-fixed" | "known-gap-not-fixed";
liveDrivable: boolean; // can this same case also run live against a real system
addedBy: string; // PR / commit / session that added it
addedAt: string; // ISO date
}
Sent to the build agent working PR 6373 / fede/after-call-poc tonight, ahead of this page, so the newest harness in the repo lands already compatible — its classRef/incidentRefs/liveDrivable/regressionStatus fields map 1:1 onto this shape with zero renaming inside its own manifest.ts.
npm run harness -- --suite promise-harness npm run harness -- --suite after-call-harness --incident matter_849f19e9 npm run harness -- --suite all --layer outcome-after-call --ci
A single entry point that reads each suite's own fixtures (through its adapter, see 3.6) and reports pass/fail + the run-attestation stamp (3.5) in one shape, regardless of which of the 45 directories actually ran. Suites keep their own internal runner (run.ts, score.ts, whatever) — the shared CLI calls into it, it doesn't replace it.
Not one grader — a registry naming which of the ~89 existing judge/grader/grade-named files is canonical for which fault class, so the next harness looks one up instead of writing #90. quality-replay/ already proves the pattern works (it imports quality-gate-poc's three-layer checker directly rather than inventing its own) — that's the model to generalize, not a new mechanism to build.
The prior inspection's "8 scrubbers → 1" instinct is wrong per Fede's own hot rule 7: a log-line masker and a committed-fixture replacer have genuinely different jobs. Two survive:
src/lib/platform/security/scrub-pii-log-string.ts (already the production module; most of the 8 should import this instead of re-deriving their own regex).piiVerified: true per row before the write happens (convert-desk-row-to-fixture.ts's existing contract). This is the harder, slower path, on purpose.All three pieces already exist in fede/after-call-poc, verified live tonight by an adversarial pass. The proposal is to make these the standard every suite adopts, not to invent something new.
fixtures.lock.json (id+suite pairs only) that a *.drift.test.ts diffs against the live manifest; a silently-deleted scenario fails loud. Proven tonight: caught a real deleted-fixture trial in the chaos/adversarial pass.{codeSha, manifestSha256, generatedAt}, so a stale or faked-green report can't be mistaken for current — if something automated checks it against the PR's actual HEAD, which today nothing does yet.scripts/mutation-controls/index.ts (enforced pre-push by scripts/check-mutation-control.ts, per Gera's decision on block b86054955) proving the guard actually reddens for its stated reason. Tonight's own finding: after-call-harness-coverage.drift.test.ts is currently missing this control and fails the check live — the newest harness in the repo isn't even compliant with its own repo's existing rule yet.Incident → fixture → red-before → green-after, one row per incident, across every suite — not per-harness bookkeeping. This is what doc-index-of-harnesses (the prior inspection's cheapest, zero-risk recommendation) becomes once it's populated from the fixture format's own provenance.incidentIds + regressionStatus fields instead of hand-maintained. No such ledger exists today; every suite tracks its own incident list in its own README, which is exactly why the same regression class gets rediscovered more than once.
| Existing harness | Keeps | Adapter does |
|---|---|---|
| promise-harness/cases.ts | Its own case shape, unchanged | Maps caseId→id, channel→subLayer, expectedPromise/expectedStatus→expected |
| after-call-harness/manifest.ts | classRef/incidentRefs/liveDrivable/regressionStatus, unchanged | Already a 1:1 rename — see 3.1 |
| Cerberus desk-row fixtures | Its own 5-file case (context/ledger/inbound/trace/expected), unchanged | Maps layer(input/loop/outbound)→subLayer, correctDisposition→expected |
| voice-harness scenario YAMLs | All 5 driver tiers, unchanged | Maps each YAML's own layer tag into subLayer under live-call |
| evals/turnover-walk (turnover Crucible) | capture-only-gauntlet.ts, false-promise-gauntlet.ts, unchanged | Wraps each gauntlet run as one HarnessFixture per scenario for the shared ledger only |
scripts/tour-decider-replay (confirmed tonight: same domain, tour-decider-replay's pure-function design is strictly safer and more complete).evals/fast-tier/ — whichever version actually runs in CI/nightly keeps the plain name, the other deletes, per suite (needs a one-time check per pair before deleting).scripts/renewal-harness/ is the live Willows-scoped rig.One page. Pin this next to the skeleton, not instead of it.
Nine systems do "replay + grade." None has everything (section 2). Picking one as the base doesn't delete the others tomorrow — it decides which plumbing new replay scenarios get written against from now on. Updated after a second, deeper pass tonight that read all 9 side by side in detail — see section 2's full comparison table.
evals/replay-harness — the ONLY one of the 9 that enters at the real production entrypoint (runPipeline→dispatchInbound→handleIncomingMessage, the same function Twilio/SQS call), runs against an audited 2,502-row/117-eligible-conversation census, grades cheapest-first (exact/tool-match, LLM judge only on divergence), and is the only one whose isolation is verified live against AWS STS at run time — independently proof-run with 0 prod writes. RECOMMENDEDquality-replay/ — real tool-execution-layer interception (recorded results served under the REAL reasoning loop) with the most rigorous prod-write isolation stack, and the only one that already reuses another suite's grader (Cerberus's checker) instead of inventing its own. Its own README calls it "spike quality," advisory/manual-trigger only, and it DOES write real (sandboxed) DDB rows — A writes none at all.Recommendation: A, with B's grader-reuse pattern folded into the shared grader registry rather than lost. Entering at the real dispatcher is the property hardest to retrofit later (it's an architectural choice, not a config flag), and A already proves the strongest isolation contract found in either system. B's genuinely valuable idea — grade by importing an existing checker instead of inventing a new one — ports cleanly into A via section 3.4's registry, so nothing gets thrown away, only re-homed.
45 directories is a lot to move at once. The G14-style check (section 4) can land immediately either way — the question is what happens to the existing 45.
Recommendation: A. The G14 check stops the bleeding immediately (no more sprawl); the adapters are additive and don't require touching a suite's own scenario files, so there's no real cost to doing them in parallel with live work instead of blocking it.
The eval deep-inspection's own finding, two days ago: "today 'the standard' lives in one person's head" — nearly every mechanism here is single-author (Fede's automation account); Gera has one commit in the escalation Crucible, zero in the coworker harness.
Recommendation: B. Naming an owner for the doctrine (not every harness — just the rule that new ones follow it) is the cheapest fix for the single-author problem; making the atlas itself generated rather than hand-maintained is what stops it becoming READMEs-with-extra-steps.