EVIDENCE.mdCerberus reads a draft reply Clara is about to send and flags specific problems — telling a resident something the tools never confirmed, Clara's internal reasoning leaking into a message, a promise nobody backed. It ships with its own CI gate that reruns it on every PR and fails the build if its accuracy gets worse. Four lanes hammered on this: regressions (plant a real bug, does the gate catch it), attacks (can a person with normal commit access fake a clean result), benign (does it stay quiet on harmless changes), and stability/ops (does it hold up under repetition and real conditions).
Hand-broke one specific piece of the checker's detection logic per trial — an inverted comparison, a dropped word, an off-by-one, a disabled short-circuit — and confirmed the gate names the exact case that starts failing.
| Trial | What was broken | What happened | |
|---|---|---|---|
| T01–T08 | Identity checks, ownership-depth search, alias lists, null-safety guards, threshold values — one break each | Every one flipped exactly the predicted case; gate named it every time | PASS ×8 |
| T09 | Broadened (not narrowed) a wording match — not expected to break anything | Correctly improved one case; gate correctly did not call it a regression | PASS |
| T09b | Forced the "did Clara claim she already did something" check to always say unbacked | Zero visible change across all 18 cases — gate reported 0 regressions | FAIL |
| T09c | Same check, opposite direction — forced it to always say grounded (permanently disables it) | Zero visible change across all 18 cases — gate reported 0 regressions | FAIL |
| T10–T15 | Narration channel guard, attribution word/phrase lists, filler-word grammar, disabled first-layer short-circuit entirely | Every one flipped the predicted case(s); largest trial flipped 6 cases, all individually named | PASS ×6 |
| AS1, AS2 | Reran two of the above breaks through the app's own vendored copy, not just the standalone checker repo | Caught identically in the app's real wiring | PASS ×2 |
Reruns, a real bug fix, a new test case, comment-only edits, a behavior-preserving refactor, and edits outside the checker's own folder. No false alarms anywhere.
| Trial | What was tried | What happened | |
|---|---|---|---|
| identical ×5 | Same code through the gate 5 times back to back | Byte-identical output every time | PASS ×5 |
| pure-improvement | Landed a real fix for a previously-known miss | Labeled "improvement," not a regression | PASS |
| add-fixture | Added a brand-new test case | Treated as new, not a deletion or regression | PASS |
| comment-only | Comments added, no logic touched | Byte-identical output | PASS |
| refactor-no-behavior | Two functions rewritten in a different style, same behavior | Byte-identical, confirmed on the larger 130-case set too | PASS |
| unrelated-file (outside folder) | Edited a file entirely outside the checker's folder | GitHub wouldn't even schedule the job | PASS |
| unrelated-file (inside folder) | Edited only the README inside the checker's folder | Job runs, correctly finds nothing to flag | PASS |
| no-baseline bootstrap | Simulated the very first PR, nothing to compare against | Correctly treated as "no baseline," not a regression | PASS |
A person with ordinary commit access tries to make a real problem look clean — editing the answer key alongside a broken checker, forging the checksum meant to catch tampering, faking a "passed" report, disabling the gate itself. Almost every real finding lives here.
| Trial | What was tried | What happened | |
|---|---|---|---|
| T01 | Deleted a test case, updated the case-count file in the same commit to hide it | Caught — the detector watches for the deletion itself, not the count | PASS |
| T02 | Broke the checker so a real case grades wrong, AND edited the answer key to match | The gate itself was fully fooled — called it an "improvement." Only a separate hand-written test caught it, and only for 15 of 18 cases | FAIL |
| T03 | Same trick, on one of 3 already-disclosed "known miss" cases with no hand-written test | Every automated check passed clean — nothing caught it | FAIL |
| T04 | Corrupted one test case's data file | Crashed the whole run loudly, no silent skip | PASS |
| T05 | Injected a realistic fake phone number and email | Caught both | PASS |
| T06 | Injected a realistic full person's name only, no phone or email | Nothing checks for names — passed clean, and shows green even though the more thorough scanner isn't configured in CI | FAIL |
| T07 | Claimed a vendored file was freshly synced, without updating its checksum | Caught — checksum drift detected | PASS |
| T08 | Edited a vendored file AND forged the checksum to match | Passed clean — it's a drift tripwire, not proof the code came from the real upstream project | FAIL |
| T09 | Injected a crash whose fallback answer happens to coincidentally be correct | Still correctly flagged as a regression — a crash is a worse signal than clean, full stop | PASS |
| T10 | Made the checker lie "all good" and write garbage output on failure | The next step in the pipeline parses the actual file content and crashes on the garbage — caught independently | PASS |
| T11 | Hand-built a fake "all 18 passed" report — never ran the real checker, including 3 known-wrong cases | Accepted as a clean, fully-passing run. Nothing confirms a report came from a real run | FAIL |
| T12 | Disabled the regression gate itself, left its checksum untouched | Caught by a separate, sibling integrity check on that exact file | PASS |
| T13 | Disabled the regression gate AND forged its own checksum in the same commit | Both the gate and its watchdog said clean, on a real, unambiguous bug. Still visible in the plain-text log — nothing forces anyone to read it instead of the checkmark | FAIL |
Flakiness, speed, cross-version behavior, cold starts, read-only permissions, and a "wrong folder" edge case that used to be a real bug.
| Trial | What was tested | What happened | |
|---|---|---|---|
| stab-01 | Same 18 cases, 20 runs back to back | Zero flip-flops across 360 individual results — fully deterministic | PASS |
| stab-02 | Timed all 20 runs | ~0.37s steady state, ~0.8s only on the very first cold run | PASS |
| stab-03 | Larger 130-case adversarial set, 5 runs back to back | Identical result every time | PASS |
| stab-04 | Looked for the "105 historical regression pairs" set named in the task | Doesn't exist anywhere in this repo or its history — only a 7-pair internal test set was found | N/A |
| stab-05 | Both Node.js versions actually used in production CI | Byte-identical results on both | PASS |
| stab-06 | Cold vs. warm dependency install | ~1s cold, ~0.8s warm — not a meaningful cost | PASS |
| stab-07 | Made the test-case folder read-only, ran anyway | Ran clean — genuinely never writes into its own test data | PASS |
| stab-08 | Ran the gate from the wrong folder (previously a real silent-pass bug) | Fails loudly with an explicit "wrong folder" error instead | PASS |
| stab-09 | (Discovered, not planted) Checked the real shared PR branch for cleanliness | Found dirty — a commit was made directly on the shared branch instead of a scratch copy, breaking the rule every lane was given. This lane confirmed it never touched that branch itself | FAIL |
Every FAIL above, ranked by severity. All 9 are gaps in the safety net around Cerberus — not in its ability to catch a real problem in a message.
One person, one commit, can turn off the regression checker and forge the separate check meant to catch tampering with it — both say "clean" on a real, obvious bug. The bug still prints in the log text, but nothing forces anyone to read it instead of trusting the green checkmark. (T13)
Nothing confirms a results file actually came from running the real checker. A hand-typed file claiming all 18 cases passed — including 3 the real checker is known to get wrong — sails through as clean. (T11)
Break the checker and edit the "correct answer" file in the same commit, and the gate doesn't just miss it — it calls the broken checker an improvement. A separate hand-written test happens to catch this for 15 of 18 cases. For the 3 cases the team has already disclosed as known misses, nothing catches it: a known gap can be silently rebranded as fixed. (T02, T03)
The "did Clara claim she already did something with no proof" check can be permanently disabled — and every one of the 18 test cases still reports the exact same result. None of the current cases actually exercise this specific check, so killing it is invisible. (T09b, T09c)
The check meant to confirm the code was really synced from the real upstream project only catches a forgotten update. Anyone who can edit the code can just as easily edit the checksum to match — a self-consistency check, not proof of where the code came from. Honestly disclosed in its own comments, but it's the same mechanism behind the HIGH finding above. (T08)
The privacy scanner never looks for names, only email/phone/SSN-shaped strings. And when the more thorough name-aware scanner isn't configured — true for every CI run today — it still shows a green checkmark. Its own log says "not checked," but a checkmark reads as "checked, clean." (T06)
Not a Cerberus defect. Someone ran a legitimate test (deleting a file to check the deletion-detector) as a real commit directly on the team's shared PR branch instead of a disposable copy, and never cleaned it up. This lane independently confirmed it never touched that branch. Needs a revert before more testing against that branch is trusted. (stab-09)
43 of 53 trials behaved exactly as they should have. 9 surfaced a real, reproducible gap — and every one of the 9 is in the safety net around the checker, not in its ability to catch a real problem.
The part that matters most — can Cerberus tell when a message has a real problem — held up everywhere it was tested head-on: 17 of 19 direct attempts to break its detection logic were caught immediately and named exactly. The 2 exceptions aren't wrong answers — they're "none of the current test cases would even notice if this one specific check went dark," a coverage gap, not a miss.
What didn't hold up: the layer meant to stop someone from gaming the gate. A person with ordinary commit access — no special privileges needed — can currently rewrite the answer key to hide a regression, fake a passing report with no real run behind it, or disable the regression check and its own tamper-detector in one commit. All three require intent, not an accident, and all three still leave a plain-text trace in the log that a human reading the actual diff would catch.
Whether that trade-off is acceptable depends on how much weight "green means safe" is expected to carry versus "a human still reads the diff" as the real backstop.