ADR-0108: Renewal pipeline — one decision spine, workflow-owned dispatch, stable per-lease identity

• Status: Accepted (Fede, 2026-07-23)
• Date: 2026-07-22
• Owner: Fede
• Context links: ADR-0047 (renewal saga as read-model), ADR-0106 (effects validate premises — the tours harmony framework this ADR extends to renewals), ADR-0091 (external-actor reconciliation), ADR-0081 (transfer guard), ADR-0092 (suppression), the 2026-07-22 saga_0a1a89ec half-start incident (PR #4318), the 2026-06-24 dropped-decline incident, the 2026-07-22 architecture review (four subsystem audits + two adversarial hunts + ~50-incident corpus + Temporal deep research, artifacts in the PR thread).

Context

The renewal pipeline is incident-hardened at the micro level but has grown by accretion into a shape where the same three structural causes keep producing incidents:

1. Duplicated rules reading divergent sources. 25 gates across 8 entry points; the signer-email rule alone is resolved four ways (scanner keys co-signers by tenant.id, the shared resolver picks the first-match occupancy by occ.id; the scanner's hasEmail requires @ while the eligibility rule accepts any non-blank string — email:"n/a" auto-sends and blind-422s). Entry points skip each other's gates: the scanner never runs the restart gate; PM routes skip the HITL verdict and full-suppression holds; no autonomous-path gate reads reasonForNonRenewal, so a soft-declined tenant with no saga can get an offer pushed to their portal at prepare time.
2. Non-transactional start sequences. Three entry points start a renewal with three different orderings of {workflow start, SQS prepare dispatch, lease stamp}, none atomic: the scanner half-starts when dispatch fails (the 2026-07-22 incident class); /outreach stamps SENT before dispatching (the exact 2026-06-23 UI-lie the scanner fixed) and strands a successorClaim when start throws; /approve dispatches SQS before starting the workflow and its catch block cites a "PR-A5 reconciler" that was never built — a Temporal outage at that moment sends a real AppFolio offer with zero PropFlow record. Scanner and PM routes can also double-mint concurrently (dedup-by-read, fresh ids, no shared claim).
3. Contested alert ownership + signal loss. The workflow's 10-minute prepare timeout classifies by string-matching (\btimeout\b passes isInfraPrepareFailure) and fires before the sweeps' live-probe classification — the dumbest signal wins (the false "infra failure" alert on saga_0a1a89ec). Meanwhile a tenant decision arriving when no live saga is findable — during the PREPARE_FAILED→retry gap, after a supersede, after terminal HOLDOVER — degrades to a "soft signal recorded on tenant only" that nothing surfaces. Closed workflow executions cannot receive signals; today's fresh-renewalId-per-retry design guarantees there are always closed executions and no stable address.

The ~50-incident corpus (2026-04 → 2026-07) distributes exactly onto these causes: signal-loss/invisible-saga 9, state-machine gaps 9, alert-misclassification 8, silent-failure/fail-open 8, rule-divergence 6, dedup-keying 5, ordering/dual-write 2 (+ external-API 7, other 7).

The Temporal layer carries its own accretion: 18 patched() guards (0 ever retired; 5 + the entire -migrated addressing shim are drained and deletable now), an inlined twin-regex error classifier kept in sync only by a drift test, three near-identical wrap-up Updates, and a 4-site copy-pasted terminal-resolution ladder. The whole transient-vs-permanent prepare apparatus exists because the workflow waits on an SQS retry loop it cannot see.

Current state (the inverted starts)

flowchart LR
    subgraph starters [Three start protocols]
        SC[Daily scanner]
        OR["/outreach (PM)"]
        AP["/approve (PM)"]
    end
    WF[renewal workflow
waits for prepared signal
never dispatches]
    SQS[(SQS prepare job
→ browser runner)]
    ROW[(Lease stamp
SENT)]

    SC -->|"1 start"| WF
    SC -->|"2 dispatch"| SQS
    SC -->|"3 stamp"| ROW
    OR -->|"1 start"| WF
    OR -->|"2 stamp (before dispatch!)"| ROW
    OR -->|"3 dispatch"| SQS
    AP -->|"1 dispatch (before start!)"| SQS
    AP -->|"2 start"| WF
    SQS -.->|"prepared / pmsFailed signal"| WF

Any single failure between steps leaves a half-state; each route chose a different damage profile; only alerts (not compensation) detect drift.

Decisions

D1 — The workflow owns its prepare dispatch (kills the half-start class)

Phase 1 of the renewal workflow calls a dispatchPmsPrepare activity as its first act, instead of every caller separately enqueuing SQS after (or before) startRenewalWorkflow. Temporal's retry policy owns retry/backoff/heartbeat; the error classifier moves to nonRetryableErrorTypes on the activity (the Lambda already classifies — pass the classification as data, deleting the inlined twin regexes and their drift test). All three entry points collapse to "gate → start workflow"; a started workflow can no longer be half-started, and /approve's SQS-orphan window (finding: PR-A5 reconciler never built) becomes unrepresentable.

The activity may still enqueue SQS for the browser runner — the point is that the enqueue happens inside the durable execution, so a failed dispatch is a retrying activity, not a stranded orchestration.

Consequences: the retry-cron's special dispatchRenewalPrepare step, the pmsFailed signal, prepare-retry-budget.ts, and 2–3 patches become deletable. Requires the prepare job's enqueue path to be callable from the worker (it already is — triggerRenewalSync is plain library code).

D2 — Stable per-lease workflow identity (kills the dropped-decline class)

Replace fresh-renewalId-per-retry addressing with a stable, deterministic workflow id per lease cycle (e.g. renewal-{leaseId}-{cycleAnchor}), and route all inbound decisions (Clara decline, NTV, PM actions) via signalWithStart / updateWithStart against that id. Temporal's documented semantics make this the atomic "start-or-signal": a decision arriving during the PREPARE_FAILED gap, after a supersede, or before any workflow exists lands in exactly one place instead of a dead letter. Retries become continueAsNew (or a fresh run under the same id), not a fresh identity — with the documented gotcha enforced: drain message handlers before every continueAsNew (allHandlersFinished), or tenant decisions are lost at the seam.

The saga row remains the read-model (ADR-0047 unchanged); precededBy chains and lease-keyed dedup (already migrated everywhere except the workflow-health sweep — fix that too) get simpler because the identity is stable. Event-history growth is bounded by continueAsNew per Temporal's 50K-event/50MB hard limit — a renewal cycle emits a handful of events, so this is hygiene, not pressure.

Interim (before D2 lands): the Clara/NTV decline finders walk the precededBy chain to the live successor instead of dropping to "soft signal on tenant only", and undeliverable decisions page the PM.

D3 — One decision spine (kills the rule-divergence class)

All gate logic funnels through one shared pipeline module returning reason codes (classifyRenewalCandidate's contract, designed for this and half-wired): an ordered RenewalGate[] consumed by scanner, /outreach, /approve, the retry preflight, and the chokepoint, with per-caller reason→disposition mapping and forceTenantId as a pipeline posture, not copy-pasted bypasses. One input-resolution per rule: co-signers keyed by the occupancy under renewal (not first-match), one email-validity predicate, one "cycle already open" implementation, and reasonForNonRenewal checked on the autonomous path (closing confirmed hunt finding #2). Fail-open gates keep their posture but count: repeated fail-open engagement raises an aggregate alarm instead of degrading silently to send-everything.

D4 — One owner per failure class (kills the alert-race class)

runPrepareRetrySweep is the sole owner of PREPARE_FAILED classification and escalation (as automations.toml already claims). The workflow's bare-timeout path stops firing renewal_prepare_infra_failure; that alert is reserved for a runner-reported failure detail. The workflow-health sweep re-keys dedup from sagaId to leaseId (the last straggler). Scanner-HELD tenants — currently the only failure class with zero push — get one deduped notifyPmPendingReview page (65-day lease-keyed, same copy module), so "flagged for human review" means a PM was actually told.

D5 — Temporal hygiene (pays down the versioning debt)

Retire the drained guards now (5 ids + the RVIEW_PROJECTION_CUTOVER date gate + the -migrated addressing shim with its double-describe() on every signal; two more ids drain 2026-07-25); adopt the documented deprecatePatch retirement flow so patches die on a schedule tied to the 60-day run-timeout instead of never. Collapse the three wrap-up Updates and the 4-site terminal ladder into one parameterized pair. Add replay testing of downloaded prod histories to CI (the repo has zero replay tests; the RVIEW wedge was a replay bug). Worker Versioning (build-id/pinned deployments) is Temporal's stated default over patched() — adopt it only if our worker topology supports long-lived versioned deployments; that check is an explicit open question, not assumed.

The measuring instrument (ratification gate)

Like the ADR-0106 bake-off, no candidate lands on judgment alone. The workflow-permutation harness (PR #4261) is extended with a pipeline domain: entry-point drivers (scanner, /outreach, /approve, retry sweep, chokepoint) run real production modules over declarative world-states with fault injection at the dual-write seams, capture dispositions + side-effect traces into a committed golden, and compute invariants (half-start, SQS-orphan, stamp-without-dispatch, dangling-claim, divergent-verdict, silent-hold). Known-bad behaviors are pinned as expectedDefect + expectedAfterFix; the diff tool classifies every candidate PR as FIXED_DEFECT / REGRESSION / UNCHANGED. The ~50-incident corpus and the 11 new hunt findings are the scenario backlog. A candidate architecture PR is accepted only when its diff shows targeted FIXED_DEFECTs, zero REGRESSIONs, and the invariant-violation count is monotonically non-increasing.

Consequences comparison

Alternatives considered

• Fix bugs individually, keep the shape. That is the last three months: ~50 incidents, each fix correct, the classes keep recurring. Rejected — the corpus is the argument.
• CDC outbox (DDB Streams → signalWithStart) instead of D1, i.e. the tours C′ shape: callers write the saga row atomically; a stream consumer starts/signals the workflow. Validated pattern (AWS prescriptive guidance + Temporal community), and it is the right shape for row-first domains like tours. Renewals are already workflow-first (ADR-0047): the workflow is the writer, so putting the dispatch inside it (D1) achieves atomicity without a new Lambda consumer to own. If D1's worker-side enqueue proves problematic, C′ is the designated fallback and is forward-compatible.
• Entity workflow spanning many cycles (one workflow per lease, forever). Cleaner addressing still, but couples every code deploy to long-lived histories and raises the versioning stakes before D5's hygiene lands. D2's per-cycle stable id + signalWithStart captures ~all of the benefit at a fraction of the risk; revisit after D5.

Research validation (2026-07-22)

Adversarially-verified deep research (106 agents, 3-vote verification per claim; full report in the session artifacts):

• Signal-With-Start is the documented atomic start-or-signal ("one total action"; client-side only — fits API-route/cron starters) (docs.temporal.io/sending-messages). Closed executions cannot receive signals — the root of the dropped-decline class; the documented mitigations are exactly D2's shapes (docs.temporal.io/encyclopedia/workflow-message-passing).
• Continue-As-New / entity workflows bound history (hard cap 51,200 events / 50MB per run) and are the recommended long-lived shape; pending signals are lost at continueAsNew unless drained first — D2 enforces allHandlersFinished (temporal.io/blog/very-long-running-workflows, docs.temporal.io/workflow-execution/continue-as-new).
• Transactional outbox via DynamoDB Streams CDC is AWS's prescriptive answer to the dual-write; at-least-once consumers must be idempotent (AWS prescriptive guidance). Kept as D1's fallback shape, consistent with ADR-0106 C′.
• Worker Versioning is Temporal's default recommendation over patched() for production code changes (pinned workflows never need patches); serverless-ish worker topologies are the stated caveat — hence D5's explicit topology check. deprecatePatch → removal after retention is the documented patch-retirement flow (docs.temporal.io/production-deployment/worker-deployments/worker-versioning, docs.temporal.io/develop/typescript/versioning).
• Replay testing of downloaded prod histories in CI is the documented pre-refactor characterization safeguard (DeterminismViolationError on incompatibility); TestWorkflowEnvironment time-skipping is the documented fast path for long-cycle tests — noted as optional here since the repo's condition-mock harness already achieves millisecond permutations (docs.temporal.io/develop/typescript/testing-suite).
• Update / Update-With-Start give synchronous read-your-writes decision delivery; Update is GA, Update-With-Start's status should be re-verified at implementation time (Public Preview as of the sourced material).

Case-study honesty: the D1 "first-activity dispatch" framing is a sound synthesis of documented primitives (activity retry ownership + atomic start-or-signal + outbox), not a single named Temporal doctrine — same epistemic status as ADR-0106's C′ synthesis.

Rollout (ratification conditions)

1. Harness first (this decision's precondition, already in flight). The pipeline domain lands with today's behavior pinned as golden, including expectedDefect annotations for every confirmed finding. No architecture PR merges without a harness diff in its body.
2. Tactical fixes that need not wait for the ADR (each currently a pinned defect; shipping them flips baselines to FIXED_DEFECT): /outreach dispatch-then-stamp reorder + transfer pre-check; /approve start-workflow-before-dispatch reorder (removes the phantom-reconciler dependency); email-predicate unification; reasonForNonRenewal gate on the autonomous path; decline-finder precededBy walk; workflow-health sweep leaseId re-key; scanner-HELD PM page. PR #4318 (pre-start signer-email gate) merges as-is.
3. D5 hygiene wave (drained-patch retirement + shim deletion + replay CI) — low-risk deletions, harness-diffed.
4. D3 spine extraction, then D1 owned dispatch, then D2 stable identity — in that order, each behind the harness gate, each proven on Willows (appfolio-45) with real lifecycles before any live-property arming, per the ADR-0106 precedent.
5. D4 alert ownership ships with D1 (the bare-timeout alert dies when the timeout-with-no-signal state dies).

What reviewers flag

• A new entry point or gate that does not go through the decision spine (post-D3), or a gate whose input resolution is re-implemented locally.
• Any caller that dispatches the PMS prepare outside the workflow (post-D1).
• Any code path that signals a bare renewalId instead of the stable per-lease address (post-D2), or a continueAsNew without a handler drain.
• A PREPARE_FAILED reactor other than the retry sweep raising an alert.
• A new patched() without a retirement date tied to the run-timeout.
• A fail-open gate without an engagement counter.
• A dedup key on sagaId/renewalId where the retry chain mints fresh ids.
• An architecture PR whose body lacks the harness diff table.