ADR-0106: Effects validate premises — Temporal timer ↔ DDB row harmony, and who owns the tour schedule

• Status: Accepted — Option C′ ratified by Fede on 2026-07-21, after the prototype bake-off (C′ #4268 vs B #4266, measured against the workflow-permutation harness #4261; results in this PR's comments). B remains the designated end-state if tours gain money/legal effects.
• Date: 2026-07-21
• Owner: Fede
• Context links: ADR-0047 (renewal saga as read-model), ADR-0052 (stuck-workflow scanner), PR #4060 (tour send integrity / reschedule signals), the tour-reminder stale-anchor guard PR (Trello BcfclCjN), 2026-07-20 phantom-reminder incident.

Context

A Temporal workflow that arms a durable timer captures a premise at arm time (e.g. "the tour is at 2026-07-20 1:00 PM, so remind at T-1h"). The entity row in DynamoDB can change after that — reschedule, cancel, out-of-band correction — and the timer's premise silently goes stale.

On 2026-07-17 a Camellia tour was created with a mis-extracted date (Mon 07-20), corrected in-app to Fri 07-17 thirteen minutes later, and toured Friday. The reschedule ran on pre-#4060 code that updated the row without signaling the workflow, so the T-1h timer kept its Monday anchor and sent a reminder for the phantom Monday slot on Mon 07-20 12:00 PM. The send activity re-read the row but only checked cancelled/already-sent; its doc comment asserted the anchor "has already been validated as reachable."

The surface root cause is state duplication with no reconciliation. The deeper root cause, surfaced in the 07-21 design review: tours have the ownership inverted relative to the renewal architecture. For renewals, the workflow is the source of truth and the DDB row is its projection (ADR-0047) — divergence is unrepresentable. For tours, the row came first (written by app CRUD from every surface) and the workflow was bolted on later as a timer side-car that copies the schedule at arm time. When the date was corrected, the code did the only thing tours have ever done — wrote the row — and the copy went stale.

Current state (the inverted shape)

flowchart LR
    subgraph writers [Many writers]
        CT[Clara schedule_tour /
reschedule_tour tool]
        UI[PM dashboard UI]
        PIPE[Email/SMS tour pipeline
apply-tour-intent.ts]
    end
    ROW[(DDB Tour row
date / startTime
SOURCE OF TRUTH)]
    WF[tourWorkflow
timer anchors =
copies of the row]
    FX[Effects: T-24h SMS,
T-1h SMS, app link]

    CT -->|write| ROW
    UI -->|write| ROW
    PIPE -->|write| ROW
    CT -.->|"signal (only since #4060,
only paths that remember)"| WF
    ROW -.->|copy at arm time| WF
    WF -->|timer fires| FX
    FX -.->|"re-read (guards:
cancelled/sent only,
pre-guard-PR)"| ROW

Properties of this shape: any forgotten signal, replay edge, or out-of-band write leaves the timer anchor stale; the effect fires on the stale premise. There are no hard gates — nothing prevents a new writer to Tour.date from appearing without the paired signal.

Decision framework — four harmony layers

New timer-driven domains pick the highest layer their risk warrants; every domain gets layer 3 unconditionally.

1. Single-writer (workflow owns the state; row is a projection)

The renewal architecture (ADR-0047). Divergence structurally impossible. Required for effects that move money, change rent, or sign documents.

2. Signal-on-write

Any code path mutating a workflow-watched field MUST signal the workflow in the same operation (the #4060 pattern). Necessary but not sufficient: it only covers writers that know to signal.

3. Verify-at-effect (mandatory, all timer-driven external effects)

Durable timers may go stale; external effects may not. Immediately before any send / PMS write / charge fired from a timer, the activity re-reads the current row and validates the premise it was armed under. Mismatch → skip with an explicit reason (skipped: 'stale-anchor') + WARN log. Prefer a version token (scheduleVersion bumped on any schedule write, captured at arm time) over value comparison. The check lives inside the effect activity (read-decide-act in one unit). Premise data in activity inputs carries dates/times/enums only — no PII (ADR-0026).

4. Drift detection (log-gated)

A periodic invariant checker comparing open workflows' premises against rows, alerting only — added for a domain only after layer-3 WARN logs show recurring stale-premise skips. Distinct from the banned compensating-cron pattern: it observes, never repairs.

The tour question: who should own the tour schedule?

Three options were evaluated. Diagrams show the write path for a reschedule.

Option A — status quo + guards (layers 2+3, shipped/shipping)

Row stays the source of truth; #4060 signals re-anchor the workflow; the stale-anchor guard makes effects validate at fire time.

flowchart LR
    W[Any writer] -->|write date| ROW[(Tour row
truth)]
    W -->|"paired signal
(by convention)"| WF[tourWorkflow]
    WF -->|timer fires| ACT[send activity]
    ACT -->|"re-read + premise check
mismatch → skip"| ROW

• Pros: already (nearly) shipped; zero migration; Temporal outage never blocks tour edits.
• Cons: the signal remains a convention — every future writer is a chance to forget; correctness rests on the last-line guard; no hard gate exists.

Option B — workflow owns the schedule (full layer 1, the renewal shape)

A tour date change is a signal (signalWithStart so in-flight or not-yet-created workflows self-create); the workflow applies it and an activity writes the row as projection. Nothing else writes schedule fields.

flowchart LR
    W[Any writer] -->|"signalWithStart
rescheduleRequested"| WF[tourWorkflow
SOURCE OF TRUTH]
    WF -->|activity writes projection| ROW[(Tour row
read-model)]
    WF -->|"timers re-anchor
atomically with the change"| FX[effects]
    RD[All readers/UI] --> ROW

• Pros: staleness unrepresentable — the timer and the schedule are the same state; one writer to reason about; matches the renewal architecture so the codebase converges on one pattern.
• Cons / consequences (the real costs):
  • Temporal enters the tour write path. Today a Temporal outage breaks reminders (fail-quiet); under B it blocks reschedules and cancellations (fail-closed). For a PM trying to move a tour while Temporal is down, that is a visible regression. Mitigable with an emergency direct-write break-glass, which then re-introduces a second writer to gate.
  • Read-your-writes UX. A signal is async; the UI must either poll the projection, or use a synchronous Temporal Update (the #3985 renewal wrap-up pattern) so the caller gets the applied result inline. Update is the right choice; it adds workflow-versioning discipline to every schedule change.
  • Migration surface. Every schedule writer must be found and rerouted: the Clara tool handlers, applyTourIntent/applyReschedule, the on-behalf re-key path (#4066/#4074), any admin/backfill scripts. In-flight tours need workflows minted (signalWithStart handles this lazily).
  • Determinism/versioning tax. Workflow code changes now require patched() discipline for in-flight tours; more history per tour (negligible storage cost, real review burden).
  • Testing. Schedule-change unit tests now need the workflow test env rather than plain repository mocks.

Option C′ — single-writer module + CDC signal delivery + hard gates (row stays truth, chokepoint enforced)

Note (07-21 research revision). Option C as first written had the chokepoint write the row and then signal the workflow in-process. The research pass found this re-opens the original bug under partial failure: if the row commit succeeds and the in-process signal call then fails (crash, Temporal unavailable, timeout), the row is re-anchored but the workflow is not — exactly the stale-anchor state of the 07-20 incident — and only the layer-3 guard catches it. So write-then-signal silently degrades to Option A under partial failure; "closes the class mechanically" was overstated. C′ closes the gap by making the chokepoint do one atomic write and delivering the re-anchor signal out-of-band via change-data-capture.

All schedule writes go through one domain function that performs a single atomic write: it writes the row and bumps scheduleVersion in one transaction — no second network call in the write path. Re-anchor signals are delivered by a DynamoDB Streams → Lambda consumer that reads the committed change and calls signalWithStart (at-least-once, idempotent on scheduleVersion). This is the transactional-outbox / CDC shape Temporal staff recommend for DB-driven workflows (community.temporal.io/t/using-external-db-events-to-drive-workflows-in-temporal/13363; AWS prescriptive guidance — transactional outbox). Hard gates make bypass fail loudly rather than drift silently.

flowchart LR
    W[Any writer] --> CHOKE["setTourSchedule()
THE one writer module"]
    CHOKE -->|"ONE atomic write:
row + bump scheduleVersion"| ROW[(Tour row
truth)]
    ROW -->|"DDB Stream (CDC)"| CON["Stream → Lambda consumer
idempotent on scheduleVersion"]
    CON -->|"signalWithStart
(at-least-once)"| WF[tourWorkflow]
    WF -->|timer fires| ACT[send activity]
    ACT -->|"version-token check
stale → skip"| ROW
    GATE1[/CI drift test:
only the chokepoint
touches date fields/] -.enforces.- CHOKE
    GATE2[/runtime: repository rejects
schedule writes without
writer authority/] -.enforces.- ROW

• Pros: closes the "forgot to signal" class mechanically and the "signal lost on partial failure" class — the signal is derived from the committed row change, not a best-effort second call, so a re-anchored row always eventually re-anchors the workflow (at-least-once, made safe by idempotency on scheduleVersion). Hard-gated in CI (drift test allowlisting one module, per the maintenance-on-temporal.drift.test.ts pattern) and at runtime (repository method requires a writer-authority token); Temporal stays off the critical write path; migration is small (reroute existing writers to the chokepoint — no workflow rewrite, no UX change); version token upgrades the fire-time guard from value comparison to mutation detection.
• Rejected alternatives (write ordering):
  • In-process write-then-signal (original C). Row commits, then the signal call fails → row re-anchored, workflow stale = the 07-20 bug recreated; guard is the only net. Rejected: the write and the signal are two separate failure domains and cannot be made atomic with DynamoDB.
  • In-process signal-first-then-write. Merely flips which side wins on partial failure (workflow re-anchored, row stale); signalWithStart cannot be enrolled in a DynamoDB transaction either, so it is no more atomic. Rejected.
• Cons: truth still lives in two places for the arm-to-fire window — the guard remains load-bearing (but now mechanical, not conventional); a determined bypass (raw DDB write outside the chokepoint) is still possible, caught only at fire time; adds one Lambda stream consumer to own and monitor (its lag is now part of re-anchor latency).

Consequences comparison

Recommendation (decision is Fede's)

Adopt C′ now; B remains the designated end-state if tours ever gain money/legal/PMS-write effects. C′ removes the two failure modes we actually observed (forgotten signal, stale premise) plus the partial-failure lost- signal hole the research pass surfaced in the original C — with hard enforcement, at low migration cost, without putting Temporal on the tour write path. Every piece of C′ (chokepoint, version token, CDC-delivered signals) is reused unchanged by a future B migration. The stale-anchor guard (in flight) ships regardless as layer-3 defense-in-depth.

Going straight to B is the fully idiomatic answer — it is the canonical Temporal Entity Workflow shape (see research below) and eliminates the dual- write problem entirely rather than managing it — if the founder accepts fail-closed reschedules during a Temporal outage (a PM cannot move or cancel a tour while Temporal is down, absent a break-glass path that itself re-introduces a second writer). C′ is the recommendation only because it keeps tour edits available during a Temporal outage; if that constraint is waived, B is strictly cleaner.

Research validation (2026-07-21)

The framework above was validated against official Temporal guidance and public engineering practice. Findings:

• Option B is the canonical shape. Temporal's own Entity Workflow pattern — one long-lived workflow per business entity, owning its state, with external changes arriving as signals/updates — is exactly Option B (temporal.io/blog/very-long-running-workflows loyalty-points example; docs.temporal.io/evaluate/use-cases-design-patterns). Our renewal architecture (ADR-0047) already is an entity workflow; B makes tours converge on it.
• Signal-driven timer re-anchoring is an official sample. The updatable-timer pattern — a durable timer that a signal cancels and re-arms — is a maintained Temporal sample (github.com/temporalio/samples-java …/updatabletimer), confirming layers 1–2 (re-anchor on signal) are idiomatic, not a workaround.
• Idempotency-by-validation backs layer 3. Temporal's guidance to make effects idempotent by validating current state at execution time, and to carry a version/idempotency token, matches our verify-at-effect + scheduleVersion design (temporal.io/blog/idempotency-and-durable-execution; github.com/joshmsmith/temporal-idempotence-by-validation).
• Update over signal-then-poll for read-your-writes. Temporal's message-passing guidance recommends Update (synchronous, returns the applied result) when the caller needs read-your-writes — validating B's UX choice of Update over polling the projection (docs.temporal.io/encyclopedia/workflow-message-passing).
• Worker Versioning GA softens B's determinism tax. Worker Versioning is now GA, letting new code pin to new workflow versions without patched() gymnastics for every change — materially reducing the versioning burden that is listed as a con of B (docs.temporal.io/worker-versioning).
• History limits are a non-issue here. Temporal's event-history size/count limits matter for high-event workflows; a tour entity emits a handful of events over its life, so B's "more history per tour" con is negligible in practice.

Case-study honesty. The pattern is heavily validated in public — Temporal's own Loyalty Points entity-workflow example, the Replay-on-Temporal conference material, and Stripe / Coinbase / Snap / Netflix as documented entity-workflow users, on the Uber/Cadence lineage Temporal descends from. But there are no public, incident-level postmortems from named companies describing a stale-timer or dual-write bug of exactly our shape. The specific DDB-Streams → Temporal signalWithStart combination in C′ is a well-supported inference from separately documented practices (transactional outbox / CDC + entity workflows), not a single cited case study — it is sound, but we should describe it as a synthesis, not as "how Company X does it."

Rollout plan (ratification conditions, Fede 2026-07-21)

C′ ships dark behind a feature flag and earns prod the hard way:

1. Willows proof first. The full C′ path (chokepoint + CDC consumer + version-token effects) runs on Willows (appfolio-45, the test property) with real tour lifecycles — book/reschedule/decline/cancel — and must demonstrably beat the current implementation on the same scenarios (reminders fire once, at the right time, for the current schedule; no phantom or suppressed sends). Evidence recorded before any live-property arming.
2. Feature flag. The C′ write path is gated (default OFF, per-property arming, fail-closed to the current path). The flag is an explicit operator control with a removal trigger: it is deleted once the prod bake completes.
3. Prod bake. After arming on the live property, C′ bakes for a couple of weeks of real traffic — stale-anchor-version skip WARNs, delivery-log rows, and the drift test all watched — before the current path is removed and the flag deleted.

What reviewers flag

• A timer-fired activity with an external effect that does not validate its premise (violates layer 3).
• Any write to Tour.date/startTime outside the schedule chokepoint (violates C′; the drift test should make this a CI failure, not a review catch).
• An in-process write-then-signal (or signal-then-write) in the chokepoint instead of one atomic write + CDC-delivered signal — reintroduces the partial-failure lost-signal hole (violates C′).
• A reschedule signal/Update handler that blocks (awaits an activity or otherwise suspends). Message handlers must be synchronous and non-blocking to stay in Temporal's guaranteed-atomic regime; blocking opens interleaving/ race windows (temporal.io/blog/robust-message-handlers).
• signalWithStart initialization logic living in the workflow main method rather than a dedicated init step — the start signal can be delivered before the main method's first run, dropping or misordering the re-anchor.
• A new write to any workflow-watched field without a paired signal (layer 2, other domains).
• A compensating cron instead of a premise check or signal.
• Premise/anchor data in activity inputs carrying PII.
• A stale-anchor skip that stamps the idempotency field as if it sent.

Consequences (framework-wide)

• Renewals: already layer 1; layer 3 still applies to timer-fired sends.
• Maintenance: dispatch re-gathers from the row at run time — layer 3 by construction; no change.
• Tours: layer 2 (#4060) + layer 3 (guard PR) now; C′'s chokepoint + CDC signal delivery + gates as the follow-up decided here; layer 4 only if guard WARN logs show recurring drift.
• Cost: one extra row read per timer fire; the chokepoint refactor for tours; one DDB-Streams Lambda consumer to own and monitor (C′); discipline cost of threading premises — paid once per domain.