The marker fail-open family

On 2026-10-24 a read-only Clara Bench replay posted at least 96 identical ops cards into a live customer's Slack in 46 seconds. The cause was not a sick database: claimTourCalendarAlert's conditional write was refused before it reached DynamoDB by a request-scoped read-only guard, which threw a plain Error. The marker's classifier tested err.name === 'ConditionalCheckFailedException' exactly, so a refusal fell through to a bare return true fail-open — and because every write inside such a scope is refused, "fail open" stopped meaning "alert once instead of going dark" and started meaning "alert once per call".

That marker was fixed in #22322, which also introduced the typed DdbWriteRefusedError and the isDdbWriteRefused helper. This page is the census of its siblings: it exists because the marker that fired was not the only one with that shape, and it is the evidence behind the slice PRs listed at the bottom.

The lead that opened this row was a grep counting bare return true lines against file-wide log calls. That cannot distinguish a fail-open from any other return true, nor tell whether a log call sits on the path that matters, so it was replaced rather than acted on.

The real instrument masks comments and string bodies at preserved offsets, brace-matches every catch (…) { }, pairs each one with its own try body, and records per site: whether the try issues a conditional write · whether the catch classifies the conditional failure · whether it rethrows · log calls inside the catch (not file-wide) · every return expression. Two masks deliberately: structure is matched with string bodies blanked, semantics with comments only — so a comment mentioning the exception does not count as a classifier, while a string literal inside a comparison does.

Swept src/ agents/ lambda/ — 5,774 files, 6,031 catch blocks, 285 catches paired with a conditional write, 27 matching the shape. Tests, *.test.ts and gitignored lambda/*/dist/ build output excluded.

Proven to discriminate, both ways, before it was trusted

Three defects the controls caught in the instrument itself

Every classifier in this family is a byte-identical copy of:

(err as { name?: string }).name === 'ConditionalCheckFailedException'

at calendar-provider-page-marker.ts:116, call-team-notice-claim.ts:93, hunt-nopickup-marker.ts:172, rating-ask-tenant-cap.ts:110, ntv-escalation-marker.ts:170, and inline at reply-not-sent-alert-marker.ts:65. Every one reads a guard refusal — or any wrapped error — as a store outage.

Each sink below was traced to the line that performs the send and then verified by hand, not inferred from a filename or a grep count.

Five of the eight fail open with no log line at all, so if it happens an operator cannot tell that it did. That, and not an imminent flood, is the strongest part of the case for fixing them.

• check_availability is in the bench door's read-only allowlist and does run for real inside the scope — but its handler, handle-check-unit-availability.ts, holds zero references to findTourSlot or calendar access. It is unit availability. The obvious route is falsified, not merely unchecked.
• No file under src/lib/domain/clara-bench references the email send or record paths (process-email-record, process-pms-email, acquireSendLock, recordReplyNotSent, sendPropertyEmail). This was the one intersection that would have recreated the flood one module over.
• Every remaining caller is a write or event path: Twilio voice routes, the maintenance rating window, the lease_states per-minute cron, a Temporal activity worker, the inbound-email Lambda.

So these fail open once per event, not once per call. None is a 96-card flood waiting to happen, and the row should not be sold to anyone as one. What remains true is the classifier: any wrapped or re-thrown error still misreads as an outage, and installDynamoMutationGuard() is a second, process-wide refusing door that wraps the onboarding-import dry run and roughly a dozen scripts.

Six files whose fail-open cannot reach a human — the cost is a duplicate internal row, a stale admin-grid pointer, or a UI value. Not worth new state.

Negative controls

A census where everything examined is a hit is not trustworthy, so two were required:

Also absent from the real census despite grep counts of 2–4: pms-records.ts, organization.ts, lead-received-outbox.ts, pms-mirror-store.ts, pms-write-intent-outbox.ts, website-leads.ts. And the transact-write twin is empty: a conditional transact throws TransactionCanceledException, and the shared helper at helpers.ts:3382 rethrows anything else, so no transact site falls through to a grant.

The MFA case is worth stating plainly because this page previously got it wrong: an outage there removes a rate brake temporarily and never locks an account, never touches enrolment, and never blocks a correct code before the fifth failure. The code check itself is Better Auth's and is unaffected. So it is not an authentication weakening, no issue was filed, and nothing is escalated. It was Q4 = no all along; only the word "weakening" overstated it.

And the sharper point for this family: every one of the eight in-scope sites documents its outage fail-open as a deliberate choice. So the in-scope list is not eight files with a bad posture — it is eight files with a correct, documented posture and a broken classifier. That is precisely why the rule below is "fix the classifier, never the posture", and why a fix that made any of them fail closed on a genuine outage would be reversing a decision somebody made on purpose and wrote down.

Each fix adds one branch before the outage arm, reusing isDdbWriteRefused rather than re-deriving it, and leaves the outage arm's return value untouched while giving it a log line. A refusal needs no bookkeeping to fail closed — nothing was posted, so there is nothing to bound — and no module-level mutable state is introduced: in #22322 a module-level bound outlived the scope it described, carried between test cases, and silently cost a planted-defect control its ability to discriminate at all.

The census found eight correct postures with one broken classifier: every site's fail-open on a store outage was deliberate and documented; the bug was that an unclassified error reached “alert”.

Then the pull request that closed the family produced three recurrences of the same shape, in our own work:

Each was caught — twice by review, once by reading a log that could have been skipped. For any assertion whose expected outcome is failure, the exit code alone is never evidence: name the failing tests, confirm a test-summary line exists at all so you know the suite started, and run the unmodified baseline in the same session — exit 0 on the branch is half the evidence.

One marker end-to-end per PR, each branched off origin/main, never off a sibling.

The instrument is a standalone script; point it at the trees and read the shape columns. Re-prove it both ways before trusting any new number — it must flag ntv-escalation-marker.ts's two sites and must not flag a catch that appears inside a comment. A count whose instrument was not challenged in both directions is a statement about the instrument, not about the codebase.