Decisions consolidated into the Clara-as-a-Coworker tracker, Phase 4 (single list) — answers there count.
Every test harness, eval rig, and replay system in the PropFlowAI codebase, in one place — plus the design for one shared skeleton so building a harness stays the house craft instead of the house sprawl.
Verified 2026-08-26 · direct find/grep/git-log against ~/.claude/propflowai (main) plus 4 named worktrees · cross-checked against the eval-quality trio (eval-inventory-2026-08-20, the eval deep inspection, the Eval & Testing Roadmap) and simplification-plan-2026-08-26 §2 bucket 4 · promptfoo layer: a separate audit is running, see promptfoo-audit.md — not redone here
The short version: at least 45 separate places in the codebase test, replay, or stress Clara's behavior — and they run on 5 different "is this actually true" answers for the same question ("did this conversation go right?"). Nine of them independently reinvent "replay a conversation and grade it." One tree of ~1,850 files isn't even in version control. A second confirmation pass tonight found four separate real-PII fixtures with zero scrubbing (not one) and the branch-protection discrepancy above. Nothing here proposes new architecture — it proposes one shared skeleton (a fixture shape, a runner, a grader list, two scrubbers, one coverage ledger, one proof-of-work stamp) that every existing harness can plug into without a rewrite, plus a doctrine and a build check so the next harness lands inside the skeleton instead of beside it.
gh api branch protectionIn plain terms: Fede loves building harnesses — that's a strength, not the problem. The problem is every harness gets built from scratch: its own way of describing a test case, its own way of running it, its own way of scrubbing private info out of it, its own way of proving the result is real and not stale. This page lists everything that exists today, shows the 9 "replay and grade" systems side by side so it's obvious which parts of which one are worth keeping, and proposes the one shared skeleton — so next time someone builds a harness, they build the SCENARIOS, not the plumbing underneath them.
Every harness/rig found tonight, one row each. Click a column header to sort; use the chips or the search box to filter. Evidence for every row: file count and last-commit date are from direct find/git log against the repo tonight; CI status is from a path-scoped grep against .github/workflows/ (not a basename substring match — several early basename hits turned out to be false positives and were corrected). Where a fact couldn't be confirmed tonight, the cell says so instead of guessing.
shown
| Name | Path | Proves | Layer | Fixture format | Runner | Grader(s) | Scrubber | Corpus | CI status | Incident origin | Overlap with |
|---|
Nine directories independently solve "load a fixture, run it through something, diff/grade the result." None has everything. This is what each has that the others lack.
| # | System | Input source | Replay mechanism | Grading | Isolation (can it write prod?) | Unique strength |
|---|
Not a rewrite. A thin, shared layer every existing harness maps onto through an adapter — the harness keeps its own scenario files, its own domain logic, its own name.
Built directly from three real formats already in the repo: promise-harness/cases.ts, after-call-harness/manifest.ts (tonight's newest harness — already has two of the three attestation mechanisms below), and Cerberus's desk-row fixture format (quality-gate-poc/scripts/convert-desk-row-to-fixture.ts, per BUILD-SPEC.md §5).
interface HarnessFixture {
id: string; // canonical id, e.g. "after-call-001"
suite: string; // owning harness, e.g. "after-call-harness", "promise-harness"
kind: "case" | "invariant"; // "case" = replay a conversation/call; "invariant" = standing
// architecture check (e.g. voice-harness's live-tool-budget)
layer: "prompt-message" | "conversation" | "outcome-after-call"
| "live-call" | "infra-ci-integrity";
subLayer?: string; // domain vocab kept AS-IS, not renamed:
// Cerberus "input"|"loop"|"outbound", voice-harness "L1".."L4"
provenance: {
source: "real-transcript" | "synthetic" | "adversarial" | "prod-replay";
sourceRef?: string; // conversationId / call id / shard:line — byte-exact pointer
incidentIds: string[]; // named incidents this fixture proves fixed
piiVerified: boolean; // required true before a real-transcript fixture can land
scrubMethod: "log-mask" | "full-replace" | "none-needed";
};
input: unknown; // opaque per-domain payload — untouched, whatever shape the
expected: unknown; // suite already uses for its case/verdict
confidence: "confirmed" | "unconfirmed";
taxonomyRefs?: string[]; // e.g. classRef[] (systematic-pattern.md classes A-M), ["R42"]
regressionStatus: "red-before-green-after" | "already-fixed" | "known-gap-not-fixed";
liveDrivable: boolean; // can this same case also run live against a real system
addedBy: string; // PR / commit / session that added it
addedAt: string; // ISO date
}
Sent to the build agent working PR 6373 / fede/after-call-poc tonight, ahead of this page, so the newest harness in the repo lands already compatible — its classRef/incidentRefs/liveDrivable/regressionStatus fields map 1:1 onto this shape with zero renaming inside its own manifest.ts.
npm run harness -- --suite promise-harness npm run harness -- --suite after-call-harness --incident matter_849f19e9 npm run harness -- --suite all --layer outcome-after-call --ci
A single entry point that reads each suite's own fixtures (through its adapter, see 3.6) and reports pass/fail + the run-attestation stamp (3.5) in one shape, regardless of which of the 45 directories actually ran. Suites keep their own internal runner (run.ts, score.ts, whatever) — the shared CLI calls into it, it doesn't replace it.
Not one grader — a registry naming which of the ~89 existing judge/grader/grade-named files is canonical for which fault class, so the next harness looks one up instead of writing #90. quality-replay/ already proves the pattern works (it imports quality-gate-poc's three-layer checker directly rather than inventing its own) — that's the model to generalize, not a new mechanism to build.
The prior inspection's "8 scrubbers → 1" instinct is wrong per Fede's own hot rule 7: a log-line masker and a committed-fixture replacer have genuinely different jobs. Two survive:
src/lib/platform/security/scrub-pii-log-string.ts (already the production module; most of the 8 should import this instead of re-deriving their own regex).piiVerified: true per row before the write happens (convert-desk-row-to-fixture.ts's existing contract). This is the harder, slower path, on purpose.All three pieces already exist in fede/after-call-poc, verified live tonight by an adversarial pass. The proposal is to make these the standard every suite adopts, not to invent something new.
fixtures.lock.json (id+suite pairs only) that a *.drift.test.ts diffs against the live manifest; a silently-deleted scenario fails loud. Proven tonight: caught a real deleted-fixture trial in the chaos/adversarial pass.{codeSha, manifestSha256, generatedAt}, so a stale or faked-green report can't be mistaken for current — if something automated checks it against the PR's actual HEAD, which today nothing does yet.scripts/mutation-controls/index.ts (enforced pre-push by scripts/check-mutation-control.ts, per Gera's decision on block b86054955) proving the guard actually reddens for its stated reason. Tonight's own finding: after-call-harness-coverage.drift.test.ts is currently missing this control and fails the check live — the newest harness in the repo isn't even compliant with its own repo's existing rule yet.Incident → fixture → red-before → green-after, one row per incident, across every suite — not per-harness bookkeeping. This is what doc-index-of-harnesses (the prior inspection's cheapest, zero-risk recommendation) becomes once it's populated from the fixture format's own provenance.incidentIds + regressionStatus fields instead of hand-maintained. No such ledger exists today; every suite tracks its own incident list in its own README, which is exactly why the same regression class gets rediscovered more than once.
| Existing harness | Keeps | Adapter does |
|---|---|---|
| promise-harness/cases.ts | Its own case shape, unchanged | Maps caseId→id, channel→subLayer, expectedPromise/expectedStatus→expected |
| after-call-harness/manifest.ts | classRef/incidentRefs/liveDrivable/regressionStatus, unchanged | Already a 1:1 rename — see 3.1 |
| Cerberus desk-row fixtures | Its own 5-file case (context/ledger/inbound/trace/expected), unchanged | Maps layer(input/loop/outbound)→subLayer, correctDisposition→expected |
| voice-harness scenario YAMLs | All 5 driver tiers, unchanged | Maps each YAML's own layer tag into subLayer under live-call |
| evals/turnover-walk (turnover Crucible) | capture-only-gauntlet.ts, false-promise-gauntlet.ts, unchanged | Wraps each gauntlet run as one HarnessFixture per scenario for the shared ledger only |
scripts/tour-decider-replay (confirmed tonight: same domain, tour-decider-replay's pure-function design is strictly safer and more complete).evals/fast-tier/ — whichever version actually runs in CI/nightly keeps the plain name, the other deletes, per suite (needs a one-time check per pair before deleting).scripts/renewal-harness/ is the live Willows-scoped rig.One page. Pin this next to the skeleton, not instead of it.
Nine systems do "replay + grade." None has everything (section 2). Picking one as the base doesn't delete the others tomorrow — it decides which plumbing new replay scenarios get written against from now on. Updated after a second, deeper pass tonight that read all 9 side by side in detail — see section 2's full comparison table.
evals/replay-harness — the ONLY one of the 9 that enters at the real production entrypoint (runPipeline→dispatchInbound→handleIncomingMessage, the same function Twilio/SQS call), runs against an audited 2,502-row/117-eligible-conversation census, grades cheapest-first (exact/tool-match, LLM judge only on divergence), and is the only one whose isolation is verified live against AWS STS at run time — independently proof-run with 0 prod writes. RECOMMENDEDquality-replay/ — real tool-execution-layer interception (recorded results served under the REAL reasoning loop) with the most rigorous prod-write isolation stack, and the only one that already reuses another suite's grader (Cerberus's checker) instead of inventing its own. Its own README calls it "spike quality," advisory/manual-trigger only, and it DOES write real (sandboxed) DDB rows — A writes none at all.Recommendation: A, with B's grader-reuse pattern folded into the shared grader registry rather than lost. Entering at the real dispatcher is the property hardest to retrofit later (it's an architectural choice, not a config flag), and A already proves the strongest isolation contract found in either system. B's genuinely valuable idea — grade by importing an existing checker instead of inventing a new one — ports cleanly into A via section 3.4's registry, so nothing gets thrown away, only re-homed.
45 directories is a lot to move at once. The G14-style check (section 4) can land immediately either way — the question is what happens to the existing 45.
Recommendation: A. The G14 check stops the bleeding immediately (no more sprawl); the adapters are additive and don't require touching a suite's own scenario files, so there's no real cost to doing them in parallel with live work instead of blocking it.
The eval deep-inspection's own finding, two days ago: "today 'the standard' lives in one person's head" — nearly every mechanism here is single-author (Fede's automation account); Gera has one commit in the escalation Crucible, zero in the coworker harness.
Recommendation: B. Naming an owner for the doctrine (not every harness — just the rule that new ones follow it) is the cheapest fix for the single-author problem; making the atlas itself generated rather than hand-maintained is what stops it becoming READMEs-with-extra-steps.