Tour-alert flood, 2026-10-24 — root cause

Read with agent_harness.slack.read_channel_history_all('C0C283G9QDR', …) over 20:00–01:00 CT.

The fail-open path logs nothing at all, so silent logs are not evidence. The only instrument that separates the candidates is the DynamoDB marker the suppressor would have written. All reads below are against propflow-prod; nothing was written, deleted or repaired.

Every org key, in one batch, with the controls inside the same batch

Western Slope resolves to org_c42d9150-b9ad-46f8-8f5f-b0fe56bacf38, but the verdict does not rest on that: production holds only 7 organizations, so all of them were swept — 7 orgs × 4 causes × 2 variants = 56 keys, plus 2 known-present ORG#…/PROFILE rows as in-batch controls.

58 keys requested
RETURNED ROWS: 2
UNPROCESSED:   {}
TOUR_CAL_ALERT rows found:        0
CONTROL rows found (must be 2):   ['ORG#org_c42d9150-…', 'ORG#org_jpco']

The two rows that came back are exactly the two controls. The instrument works; the marker rows are absent.

Every Western Slope property key

90 properties (read from proporg-index) × 4 key variants, in four batches, each batch again carrying a known-present control row. Every batch: per-property-markers=0, control=1, unprocessed=False.

Why absence here means "never written" and not "expired"

• The marker's re-arm window is 24 h and its ttl is expiresAt/1000 — a row written at 21:56 CT expires 2026-10-25 21:56 CT.
• The query ran at 22:34 CT, 38 minutes after the flood, well inside that window. DynamoDB TTL deletion is also best-effort and lags, which can only make a row more likely to persist, never less.
• Nothing in the codebase deletes a TOUR_CAL_ALERT# row.

A successful conditional put would have left exactly one row and suppressed cards 2…96. Zero rows beside 96 cards means every put failed without raising ConditionalCheckFailedException — which is the fail-open branch, by definition.

A complete census of production requests during the flood minute (below) shows the traffic was almost entirely POST /api/internal/bench/context. That route's prospect-email path — added by #22238, which also wrapped it in #22201's scope — runs its whole turn build inside that scope:

POST /api/internal/bench/context   { prospectEmail: … }
  → prospect-email.ts:135      withDdbReadOnly(() => buildTurn(ref))
  → buildTurn:169              readGuestCardEngineTurnInputs
  → guest-card-home-facts-fetch.ts:129  readGuestCardHomeFacts
  → :177                       fetchFirstEmailTourSlots(propertyId)
  → first-email-tour-slots.ts  findTourSlot
  → check-availability.ts      checkAvailability → alertDroppedPoolHosts (:1819)
  → tour-calendar-ops-alert.ts notifyTourCalendarReadFailed
  → tour-calendar-alert-marker.ts  claimTourCalendarAlert → PutCommand
  → ddb-read-only-scope.ts:49  throw new Error('DynamoDB write refused: … It did not reach DynamoDB.')
  → isConditionalCheckFailed(err) === false      // err.name is 'Error'
  → return true                                  // FAIL OPEN
  → postOpsEvent → Slack card

Two things that had to be true for this to produce a card, and both are

• Reads had to keep working. The WHO cell resolved in all 96 cards, and that runs getPersonsByIds → batchGetRereadingOnce, which uses the same TABLE_NAME constant and the same doc client as the refused write. Reads pass the guard; writes do not. Same invocation, opposite outcomes — which also rules out a wrong or missing DYNAMODB_TABLE_NAME.
• The card had to survive a blanket write refusal. postOpsEvent performs only reads before its Slack call at ops-event.ts:683; its one DynamoDB write (the conversation thread anchor) happens after the post and deliberately swallows its own errors. So nothing on the posting path needed a successful write.

A complete request census was taken for 02:56:00–02:56:56 UTC in seven short slices, each returning well under the API's 50-row cap so that each slice is exhaustive rather than a truncated tail.

• 87 of 90 requests were bench-door traffic. Ordinary customer traffic in that minute was three requests, none of them leasing.
• One cause fired, not two, so the ceiling is ~5 claims per findTourSlot walk (MAX_DAYS = 5), not ~10. And host_dropped means other hosts still had slots, so the walk resolves early — in practice about one claim per request.
• Of the 87 bench requests, 50 returned 200; the 19 404s and 18 422s are refs that never resolved, so they almost certainly never reached buildTurn. ≥96 cards over ~50 successful builds is ≈2 cards per build — consistent with fetchFirstEmailTourSlots wanting a morning and an afternoon slot and walking a second day. This was not one walk: it was dozens of separate serverless invocations, each making its own claim.
• Therefore the fix must hold across processes. An in-memory or per-request dedupe would not have stopped any of this. The durable DynamoDB marker was the right design; it was disabled, not inadequate.
• Every read target in those logs is propflow-prod, and all 90 requests served one deployment on branch release, created 21:47:05 CT and Ready roughly four minutes before the first card.

• The fail-open itself logs nothing — by construction. claimTourCalendarAlert returns true on a store error with no logWarn. CloudWatch and Sentry silence on candidate (b) is expected and carries no information either way.
• Sentry was not queried. No conclusion should be drawn from its absence here.
• No complete census exists outside 02:56:00–02:56:56 UTC. The Vercel log API caps a response at 50 rows and ignores --limit; a wide window silently returns the most recent 50, which is a tail. Only short windows are exhaustive.
• The bench runner's own refusedWrites report was not retrieved — it is returned to the caller (repo clara-bench), not persisted here. Note too that the alert is fire-and-forget, so a refused write can land after refusedWrites is read; a report that omits it would not be exoneration.
• Gera's "Why are you spamming these, stop it now" was not located in this channel. That is a statement about the search, not about whether he said it — it may be older, in a thread, or elsewhere. This page does not assert where it was said.

Instrument failures hit along the way, recorded so the bad numbers are not trusted

• vercel logs --limit 1000 returned 50 rows. The cap is 50 and the flag is ignored — a count from one unsliced call is meaningless.
• A first attempt at slicing used set -- $W; zsh does not word-split, so every slice received one malformed timestamp and returned 0 rows. That zero was my own bug, and it looked exactly like "no traffic". The slices were redone with explicit variables, and the one-minute window served as the control that proved traffic existed.
• ItemCount on a table or index was never used as evidence; every marker result above is a key-level read.

Also: tour-calendar-ops-alert.test.ts carries 14 cases, all sequential, and not one with the claim store erroring. The existing suite was never going to catch this.