Cerberus Burn-In

Extreme testing before dialing on: 53 hand-built trials across four independent lanes, trying to break the checker itself and the CI safety net wrapped around it, before anyone relies on it.
2026-08-21 · app PR #6039 · full trial-by-trial detail: EVIDENCE.md
53
Trials run
43
Behaved correctly
9
Real findings
1
Could not run
Can it still catch a real problem?
17 / 19 hand-planted bugs in the checker's own logic were caught and named exactly.
Can someone fool the safety net around it?
7 / 13 deliberate attempts to fake a clean result were caught. 6 got through.

Cerberus reads a draft reply Clara is about to send and flags specific problems — telling a resident something the tools never confirmed, Clara's internal reasoning leaking into a message, a promise nobody backed. It ships with its own CI gate that reruns it on every PR and fails the build if its accuracy gets worse. Four lanes hammered on this: regressions (plant a real bug, does the gate catch it), attacks (can a person with normal commit access fake a clean result), benign (does it stay quiet on harmless changes), and stability/ops (does it hold up under repetition and real conditions).

Lane 1 · Regressions 17/19 pass

Hand-broke one specific piece of the checker's detection logic per trial — an inverted comparison, a dropped word, an off-by-one, a disabled short-circuit — and confirmed the gate names the exact case that starts failing.

TrialWhat was brokenWhat happened
T01–T08Identity checks, ownership-depth search, alias lists, null-safety guards, threshold values — one break eachEvery one flipped exactly the predicted case; gate named it every timePASS ×8
T09Broadened (not narrowed) a wording match — not expected to break anythingCorrectly improved one case; gate correctly did not call it a regressionPASS
T09bForced the "did Clara claim she already did something" check to always say unbackedZero visible change across all 18 cases — gate reported 0 regressionsFAIL
T09cSame check, opposite direction — forced it to always say grounded (permanently disables it)Zero visible change across all 18 cases — gate reported 0 regressionsFAIL
T10–T15Narration channel guard, attribution word/phrase lists, filler-word grammar, disabled first-layer short-circuit entirelyEvery one flipped the predicted case(s); largest trial flipped 6 cases, all individually namedPASS ×6
AS1, AS2Reran two of the above breaks through the app's own vendored copy, not just the standalone checker repoCaught identically in the app's real wiringPASS ×2

Lane 2 · Benign changes 12/12 pass

Reruns, a real bug fix, a new test case, comment-only edits, a behavior-preserving refactor, and edits outside the checker's own folder. No false alarms anywhere.

TrialWhat was triedWhat happened
identical ×5Same code through the gate 5 times back to backByte-identical output every timePASS ×5
pure-improvementLanded a real fix for a previously-known missLabeled "improvement," not a regressionPASS
add-fixtureAdded a brand-new test caseTreated as new, not a deletion or regressionPASS
comment-onlyComments added, no logic touchedByte-identical outputPASS
refactor-no-behaviorTwo functions rewritten in a different style, same behaviorByte-identical, confirmed on the larger 130-case set tooPASS
unrelated-file (outside folder)Edited a file entirely outside the checker's folderGitHub wouldn't even schedule the jobPASS
unrelated-file (inside folder)Edited only the README inside the checker's folderJob runs, correctly finds nothing to flagPASS
no-baseline bootstrapSimulated the very first PR, nothing to compare againstCorrectly treated as "no baseline," not a regressionPASS

Lane 3 · Attacks 7/13 pass

A person with ordinary commit access tries to make a real problem look clean — editing the answer key alongside a broken checker, forging the checksum meant to catch tampering, faking a "passed" report, disabling the gate itself. Almost every real finding lives here.

TrialWhat was triedWhat happened
T01Deleted a test case, updated the case-count file in the same commit to hide itCaught — the detector watches for the deletion itself, not the countPASS
T02Broke the checker so a real case grades wrong, AND edited the answer key to matchThe gate itself was fully fooled — called it an "improvement." Only a separate hand-written test caught it, and only for 15 of 18 casesFAIL
T03Same trick, on one of 3 already-disclosed "known miss" cases with no hand-written testEvery automated check passed clean — nothing caught itFAIL
T04Corrupted one test case's data fileCrashed the whole run loudly, no silent skipPASS
T05Injected a realistic fake phone number and emailCaught bothPASS
T06Injected a realistic full person's name only, no phone or emailNothing checks for names — passed clean, and shows green even though the more thorough scanner isn't configured in CIFAIL
T07Claimed a vendored file was freshly synced, without updating its checksumCaught — checksum drift detectedPASS
T08Edited a vendored file AND forged the checksum to matchPassed clean — it's a drift tripwire, not proof the code came from the real upstream projectFAIL
T09Injected a crash whose fallback answer happens to coincidentally be correctStill correctly flagged as a regression — a crash is a worse signal than clean, full stopPASS
T10Made the checker lie "all good" and write garbage output on failureThe next step in the pipeline parses the actual file content and crashes on the garbage — caught independentlyPASS
T11Hand-built a fake "all 18 passed" report — never ran the real checker, including 3 known-wrong casesAccepted as a clean, fully-passing run. Nothing confirms a report came from a real runFAIL
T12Disabled the regression gate itself, left its checksum untouchedCaught by a separate, sibling integrity check on that exact filePASS
T13Disabled the regression gate AND forged its own checksum in the same commitBoth the gate and its watchdog said clean, on a real, unambiguous bug. Still visible in the plain-text log — nothing forces anyone to read it instead of the checkmarkFAIL

Lane 4 · Stability & ops 7/9 pass, 1 n/a

Flakiness, speed, cross-version behavior, cold starts, read-only permissions, and a "wrong folder" edge case that used to be a real bug.

TrialWhat was testedWhat happened
stab-01Same 18 cases, 20 runs back to backZero flip-flops across 360 individual results — fully deterministicPASS
stab-02Timed all 20 runs~0.37s steady state, ~0.8s only on the very first cold runPASS
stab-03Larger 130-case adversarial set, 5 runs back to backIdentical result every timePASS
stab-04Looked for the "105 historical regression pairs" set named in the taskDoesn't exist anywhere in this repo or its history — only a 7-pair internal test set was foundN/A
stab-05Both Node.js versions actually used in production CIByte-identical results on bothPASS
stab-06Cold vs. warm dependency install~1s cold, ~0.8s warm — not a meaningful costPASS
stab-07Made the test-case folder read-only, ran anywayRan clean — genuinely never writes into its own test dataPASS
stab-08Ran the gate from the wrong folder (previously a real silent-pass bug)Fails loudly with an explicit "wrong folder" error insteadPASS
stab-09(Discovered, not planted) Checked the real shared PR branch for cleanlinessFound dirty — a commit was made directly on the shared branch instead of a scratch copy, breaking the rule every lane was given. This lane confirmed it never touched that branch itselfFAIL

Findings, plain English

Every FAIL above, ranked by severity. All 9 are gaps in the safety net around Cerberus — not in its ability to catch a real problem in a message.

One coordinated commit can defeat the whole safety net at onceHIGH

One person, one commit, can turn off the regression checker and forge the separate check meant to catch tampering with it — both say "clean" on a real, obvious bug. The bug still prints in the log text, but nothing forces anyone to read it instead of trusting the green checkmark. (T13)

A fake "everything passed" report is accepted with zero proof it came from a real runHIGH

Nothing confirms a results file actually came from running the real checker. A hand-typed file claiming all 18 cases passed — including 3 the real checker is known to get wrong — sails through as clean. (T11)

The answer key can be quietly edited to match a broken checker's wrong answerHIGH

Break the checker and edit the "correct answer" file in the same commit, and the gate doesn't just miss it — it calls the broken checker an improvement. A separate hand-written test happens to catch this for 15 of 18 cases. For the 3 cases the team has already disclosed as known misses, nothing catches it: a known gap can be silently rebranded as fixed. (T02, T03)

A whole detection check can be killed with zero warningHIGH

The "did Clara claim she already did something with no proof" check can be permanently disabled — and every one of the 18 test cases still reports the exact same result. None of the current cases actually exercise this specific check, so killing it is invisible. (T09b, T09c)

The vendored-code checksum only catches accidents, not someone doing it on purposeMEDIUM

The check meant to confirm the code was really synced from the real upstream project only catches a forgotten update. Anyone who can edit the code can just as easily edit the checksum to match — a self-consistency check, not proof of where the code came from. Honestly disclosed in its own comments, but it's the same mechanism behind the HIGH finding above. (T08)

A realistic person's name with no phone or email isn't checked for at allMEDIUM

The privacy scanner never looks for names, only email/phone/SSN-shaped strings. And when the more thorough name-aware scanner isn't configured — true for every CI run today — it still shows a green checkmark. Its own log says "not checked," but a checkmark reads as "checked, clean." (T06)

The shared work copy was left dirty by a process slipOPERATIONAL

Not a Cerberus defect. Someone ran a legitimate test (deleting a file to check the deletion-detector) as a real commit directly on the team's shared PR branch instead of a disposable copy, and never cleaned it up. This lane independently confirmed it never touched that branch. Needs a revert before more testing against that branch is trusted. (stab-09)

Not a finding — a gap in the ask: the "105 historical regression pairs" named in the task don't exist anywhere in this repo or its history. Only a 7-pair set used for internal unit tests was found. If the 105-pair set lives elsewhere, point us at it. (stab-04)

The honest bottom line

43 of 53 trials behaved exactly as they should have. 9 surfaced a real, reproducible gap — and every one of the 9 is in the safety net around the checker, not in its ability to catch a real problem.

The part that matters most — can Cerberus tell when a message has a real problem — held up everywhere it was tested head-on: 17 of 19 direct attempts to break its detection logic were caught immediately and named exactly. The 2 exceptions aren't wrong answers — they're "none of the current test cases would even notice if this one specific check went dark," a coverage gap, not a miss.

What didn't hold up: the layer meant to stop someone from gaming the gate. A person with ordinary commit access — no special privileges needed — can currently rewrite the answer key to hide a regression, fake a passing report with no real run behind it, or disable the regression check and its own tamper-detector in one commit. All three require intent, not an accident, and all three still leave a plain-text trace in the log that a human reading the actual diff would catch.

Whether that trade-off is acceptable depends on how much weight "green means safe" is expected to carry versus "a human still reads the diff" as the real backstop.

Merge remains Fede's call. This evidence pack is the input, not the decision.
PropFlow Docs