0052 — Temporal deploy safety: graceful drain, replay gate, Pinned (Worker Deployments), and a no-fork migration SOP

• Status: Proposed
• Date: 2026-06-09
• Deciders: Fede
• Extends: ADR-0025 — adopt Temporal Cloud for renewal workflows. This ADR does not reverse 0025; it hardens how we deploy, version, address, and onboard workflows on the engine 0025 chose.
• Related: ADR-0026, ADR-0045 (namespace consolidation — a pending in-flight migration that must use this ADR's SOP), ADR-0046, ADR-0047
• Trello: #1027 NTV handler not flipping declined renewals (the pre-existing bug this closes, open since 2026-05-08), #1196 unify namespaces → propflow-prod / ADR-0045, #1073 Temporal Cloud provisioning
• Incident status: Resolved 2026-06-09 via the Decision §5 fix-forward path — dispatch guard (PR #2115) deployed first, terminal signals pre-buffered into the wedged executions, then the replay-compatibility gate + blocking CI replay corpus (PR #2116) deployed; both remaining wedged workflows resumed in place and closed with zero tenant sends.
• Note: docs/adr/ currently has a duplicate 0048- number (pre-existing); 0052 is the correct next number — do not propagate the collision.

Context

ADR-0025 adopted Temporal Cloud to make renewals "bulletproof": durable timers and signals replace the saga-watchdog cron. It flagged that "workflow versioning during deploys requires patched()" and that "CI lints for missing patched() calls." That control proved insufficient. On 2026-05-27 a production incident wedged 3 renewal workflows, froze the renewals read model since 2026-05-20, and fired no alert. Five Camellia tenants who filed Notice-to-Vacate had renewals that never advanced to declined.

The incident. Three renewals were migrated from pre-existing sagas on 2026-05-18 (PR #1080) via scripts/temporal/migrate-sagas-to-workflow.ts, which starts a workflow at workflowId=renewal-<sagaId>-migrated that loads its resume state through the recorded loadSagaInitialState Phase-0 activity. Root cause (confirmed 2026-06-09 by replaying the exported prod histories locally at candidate commits): the migrated histories are well-formed — they recorded all three patch markers (phase0_load_saga_state_v1, letter_mms_in_workflow_v1, outreach_parallel_v2) exactly as the 05-18 worker wrote them. What broke them is PR #1214 (2026-05-20, RVIEW projection cutover), which added a recordWorkflowPhase activity inside setPhase() with no patched() guard. Replaying any pre-#1214 history, the post-#1214 worker suspends on that unexpected activity mid-activation and never reaches the patched('letter_mms_in_workflow_v1') call recorded in the same activation — so the marker has no matching change command: [TMPRL1100] Nondeterminism error: Non-deprecated patch marker encountered for change "letter_mms_in_workflow_v1", but there is no corresponding change command!. (The error names #1033's marker, which misdirected the first diagnosis toward the migration's history shape; the marker is merely where the misalignment surfaces.) Any pre-#1214 in-flight workflow would have wedged on its next replay — the migrated 3 were simply the only ones still open when their 7-day reminder timers fired on 05-27. The worker — deployed under the no-drain --force-new-deployment model — has retried that workflow task forever since; temporal workflow describe shows ExecutionStatus=Running, StateTransitionCount ~4310 vs HistoryLength 55.

Three stacked failure modes — the root cause is structural, not a one-off.

1. No deploy safety. The worker runs unversioned (useVersioning defaults false; effective build id unversioned) and ECS --force-new-deployment has no graceful drain (stopTimeout: 30, no SIGTERM handler). An in-flight workflow's next task is served by new code immediately → new code meets old/synthetic history → patched() nondeterminism → wedge.
2. No pre-merge determinism gate, and long timers hide it. There is no runReplayHistories test in the repo; the only check is the nightly replay-verifier.ts, which is continue-on-error: true. A non-deterministic change ships green and stays invisible until a 7/14/30-day timer fires days later — then wedges. Silent until too late.
3. A migration that amplifies the blast radius — plus two regressions it caused.
   • Forked history: workflowIdOverride=renewal-<sagaId>-migrated produces a second id space and a cohort whose history shape diverges from greenfield runs over time. The fork did not cause the marker mismatch (that was #1214's unguarded activity) — but it created a long-lived cohort nobody's replay thinking accounted for, and it broke addressing and the terminal flag below.
   • Addressing (F2): because migrated workflows live at renewal-<sagaId>-migrated but every client.ts signal/query helper targets the bare renewalId, signals miss the real workflow — and with startIfMissing, signalWithStart spawns a ghost empty workflow at the bare id.
   • Terminal flag (F3): the migration set saga.closedAt on MIGRATED_TO_WORKFLOW; findOpenSagaForTenant gates on closedAt === null, so the NTV-decline handler silently skipped migrated tenants (only stamping reasonForNonRenewal). This is the bug already ticketed as Trello #1027 on 2026-05-08.

No Sentry alert fired for any of this: Temporal ReportedProblems are not forwarded to Sentry, and the only verifier is nightly and non-blocking.

Namespace context (relevant to the deploy/migration story). Five Temporal namespaces exist on account hmqma: propflow-renewal-prod.hmqma (renewal + tour + turnover today — where the wedged workflows live; 30-day retention), propflow-renewal-stage.hmqma, propflow-prod.hmqma (maintenance; ADR-0045's target), propflow-tools-prod.hmqma (Agent Smith), and a fail-open default. ADR-0045 (Trello #1196, not started) plans to consolidate the tenant backend onto propflow-prod. A namespace move is itself a migration of in-flight executions across namespaces — done with today's fork-history pattern it would wedge identically. ADR-0045 therefore depends on the no-fork SOP defined here. Recovery and the fixes below target the current propflow-renewal-prod.hmqma.

Decision

A layered deploy-safety and migration model. Fundamentals first; Worker Deployments / Pinned as the structural backstop once the cheaper, higher-leverage fixes are in.

1. Workflow-id & addressing convention — retire -migrated

workflowId === renewalId, always — for greenfield and onboarded executions alike. The -migrated suffix is removed. It was introduced to avoid colliding with a greenfield start at the same id, but WorkflowIdConflictPolicy.USE_EXISTING already makes that safe, so the suffix bought nothing and broke signal/query addressing and the cross-system audit-log join. One id space restores correct addressing and means a workflow's history shape no longer depends on how it was created.

• Cohort-drain addressing fallback (until the existing -migrated cohort drains): client.ts signal/query helpers do a no-start resolve-then-signal probe — describe both renewalId and renewal-<id>-migrated, signal the one that is running; never signalWithStart during the fallback (the USE_EXISTING + ALLOW_DUPLICATE combo would re-create the ghost-workflow bug). Delete this addressing fallback once no -migrated executions remain.

2. Deploy safety — now

• Graceful drain. renewal-worker.ts handles SIGTERM/SIGINT with worker.shutdown() (stop polling, finish in-flight tasks, resolve run()), shutdownGraceTime '25s'; raise ECS stopTimeout 30→120. Keep minimumHealthyPercent=100,maximumPercent=200 so the new task boots in parallel before the old drains. (Honest scope: drain only closes the healthy-handoff window — it does not rescue a forever-retrying nondeterministic task; the replay gate is the real "now" defense for the F1 class.)
• Blocking pre-merge replay-determinism gate. A vitest test runs Worker.runReplayHistories() over a corpus of real exported prod histories (greenfield mid-outreach, timer-blocked, and a -migrated example) against the current bundle, on every PR; the nightly verifier's determinism assertion becomes blocking. Auto-export every newly-migrated/onboarded workflow's history into the corpus as a migration step — otherwise the gate cannot cover a cohort created after the snapshot, which is exactly the #1033→#1080 gap. All corpus entries must be verified against ADR-0026's PII boundary (activity inputs carry only (conversationId, runId) + counts/enums, never phone/email/tenant names) or passed through a PII scrub before committing to the repo; pre-ADR-0026 histories should be regenerated via a greenfield replay rather than exported from prod — activity results (e.g., loaded saga state) are broader than the ADR-0026-clean inputs and may embed tenant details, and the -migrated example is the highest risk since it predates stricter gate enforcement.
• Patch discipline. docs/runbooks/temporal-patch-lifecycle.md (alongside the existing runbooks — no new top-level docs/temporal/ directory) records each changeId with introduce / deprecate-after / remove dates. patched() may only guard a genuinely new branch on a path a pre-patch history never executed. The deprecate-after date keys off max(workflowRunTimeout, longest outstanding timer horizon), not a flat 60 days; the replay gate (not the calendar) is what actually authorizes removal.

3. Migration & onboarding SOP — never fork history

• Onboard at workflowId === renewalId (no suffix).
• The workflow loads its state from DDB via the recorded loadSagaInitialState activity, not synthetic in-memory injection. Un-patch-gate Phase-0 (deprecatePatch → remove, each step replay-verified) and delete the migratedFrom resume branch once unreferenced.
• Seed receipts first (reconstructReceipts); the (renewalId, activityName) receipt store is the only cross-run idempotency backstop (the per-call (workflowId, runId, …) key is run-scoped and gives zero cross-run protection — a fact the recovery runbook depends on).
• Stop setting closedAt on MIGRATED_TO_WORKFLOW (drop migrate-sagas-to-workflow.ts:645); it is a handoff, not a terminal — leave closedAt: null until a real terminal.
• Never deploy a new patched() marker between saga creation and migration completion (the rule #1033→#1080 violated).
• Cross-namespace migration (ADR-0045) uses this same SOP — fresh executions in the target namespace that load state from the DB; never replay a foreign-namespace history.

4. Worker Deployments + Pinned — later, structural backstop

• SDK 1.17.2 deprecates the term "Worker Versioning"; the live API is Worker Deployments. Migrate off the deprecated buildId+useVersioning to workerDeploymentOptions: { useWorkerVersioning: true, version: { deploymentName: 'renewal-worker', buildId: <git-sha> }, defaultVersioningBehavior: 'PINNED' } with setWorkflowOptions({ versioningBehavior: 'PINNED' }, renewalWorkflow); short scan/cohort workflows use AUTO_UPGRADE so a cohort scanner doesn't hold the deployment draining. (Field names must be verified against the @temporalio/worker version pinned in package.json at implementation time — the API shape is still evolving.)
• Rework deploy-renewal-worker.yml to run two versions during cutover and temporal worker deployment set-current-version (CLI/gRPC — the 1.17.2 TS client has no deployment-management handle) with a describe health gate, instead of --force-new-deployment.
• Pinned is the only complete protection for a workflow idle on a multi-day timer during a deploy. Cost / prerequisite: with workflowRunTimeout: 60 days, Pinned means keeping each prior build pollable for up to 60 days — potentially many concurrent live builds on an auto-deploy service. Continue-As-New is therefore a prerequisite of affordable Pinned, not an optional follow-up (it shrinks the pin window). This is the main reason for the now-vs-later sequencing.

5. Recovery — fix-forward first; terminate is the last resort

A workflow wedged on nondeterminism is not dead: its workflow task fails and retries forever, and every completed activity remains in history. The recovery hierarchy, best → last resort:

1. Fix-forward (primary; zero side effects). Make the current worker code replay the wedged history without diverging — reproduce or gate the recorded decision points (patched() markers, activity schedules) so the prefix replays exactly; validate with Worker.runReplayHistory against the exported history locally (replay executes no activities and touches no network). Deploy. The next task retry succeeds and the workflow resumes in place: completed activities are not re-run, so nothing re-sends. This is how the 2026-05-27 wedge was resolved (2026-06-09), in this order: (a) the no-comms-to-declined dispatch guard (PR #2115) merged and its worker deployed first; (b) terminal signals pre-buffered into the two still-wedged -migrated executions (see the pre-buffer rule below); (c) the start-time replay gate + blocking CI replay corpus (PR #2116) merged and deployed — both workflows resumed in place (same runId), consumed their buffered signals before any timer, and closed declined/executed with zero tenant sends. Diagnose against the exported history before choosing a recovery path — temporal workflow show -o json + the local replayer pins the exact divergence in minutes.
2. temporal workflow reset (fallback; needs replay-compatible code anyway). Rewinds to a prior workflow task; activities completed before the reset point are preserved. Use --reapply-type None so buffered signals are not re-delivered (signal-driven side effects don't double-fire). Only useful when the divergence point itself must be discarded — and only after the code can replay the surviving prefix, otherwise it re-wedges.
3. Terminate + fresh-start at the bare renewalId (last resort; destructive). Discards the execution and re-runs from scratch — every activity re-fires unless suppressed. Requires the receipt-completeness gate (reconstructReceipts dry-run confirming a RENEWAL_RECEIPT#<sagaId> row per already-sent channel) and the activity-level no-comms-to-declined guard before any signalWithStart. Reserve for histories that are truly unrecoverable (corrupted, or semantically wrong to resume). The cost is real: the 2026-06-09 first recovery used this path on a wedged renewal whose tenant had a Notice to Vacate on file, and re-dispatched her renewal-offer letter (unit 414) minutes before the decline signal landed.

Pre-buffer terminal signals before deploying the fixed worker whenever the workflow's real-world outcome is already decided. Signals sent to a wedged (still-Running) execution buffer durably in history; when the fixed worker picks up the task, the SDK delivers buffered signals at the start of the activation, ahead of any pending timer — so the terminal exit wins every race against a queued reminder. The signal-after-resume ordering lost exactly that race once (a renewal-offer letter re-dispatched minutes before the decline signal landed); pre-buffering is the proven, race-free pattern, rehearsed live on the Test Property first (REHEARSAL_BUFFERED_SIGNAL=<state> mode of scripts/temporal/rehearse-wedge-fix-forward.ts). The terminal signal by outcome:

• NTV on file → signalPmsChanged({to:'ntv'}) (the Phase-agnostic early exit; closes declined) — not signalTenantIntent('vacate') (only honored at Phase 3 — it would walk the outreach ladder first). This closed the NTV tenant's workflow in the 2026-06-09 recovery.
• Renewal already countersigned in the PMS → signalPmsChanged({to:'countersigned'}) (closes with outcome executed). PMSState has no 'renewed' value — countersigned is the correct terminal signal. This closed the countersigned tenant's workflow in the 2026-06-09 recovery.

6. Observability — now

• Primary (metrics path): the workflow sandbox cannot call Sentry directly; scrape the Core SDK metric temporal_workflow_task_execution_failed{failure_reason~nondeterminism} and alert to Sentry level:fatal, tag source=temporal, fingerprint by workflowId.
• Near-real-time scanner: promote replay-verifier.ts to a scheduled job (~10 min) that flags ReportedProblems, the StateTransitionCount-vs-HistoryLength divergence, and pending-activity attempt>5 → Sentry. Honest SLA: "within one scan interval (~10 min)." (Vercel crons only fire on prod — Trello #1097 — so this scanner runs as an EventBridge/Temporal Schedule per ADR-0003, not a Vercel cron; and it must read the same DDB table the workers write.)
• Make signalNtvIntent's silent skip a Sentry event, not logInfo. Sentry, never Telegram (team convention).

Consequences

Easier: new code never meets old in-flight history unsafely (drain now, Pinned later); non-deterministic changes fail a PR not a customer renewal; one workflow-id space + DB-backed startup means one history shape, so patched() stops being migration-fragile and signal addressing is correct; a wedge pages within one scan interval instead of staying silent for a week; ADR-0045's namespace move inherits a safe migration path.

Harder: a recorded-history corpus must be maintained and grown (and auto-exported on migration); Worker Deployments add operational weight (two versions live during cutover, build retention up to the run timeout, CLI/gRPC promotion, drain monitoring); the try-both-ids addressing fallback is temporary debt to be deleted; per-marker deprecatePatch lifecycle is human-enforced.

Follow-ups: docs/runbooks/temporal-patch-lifecycle.md; docs/runbooks/temporal-stuck-workflow-recovery.md (fix-forward-first per Decision §5; the local replay tooling already exists — scripts/temporal/replay-history.ts, scripts/temporal/scrub-history-fixture.ts, and the blocking renewal-workflow-replay-corpus.test.ts vitest gate); scripts/temporal/recover-wedged-renewal.ts re-scoped to the last-resort terminate path only; the Test-Property wedge/fix-forward rehearsal harness now exists (scripts/temporal/rehearse-wedge-fix-forward.ts — used live before the 2026-06-09 recovery) and should be extended to rehearse migration as well; Continue-As-New promoted from "nice to have" to a prerequisite of Pinned.

What went wrong / lessons

• One unguarded activity inside a shared helper wedged every pre-deploy in-flight workflow. PR #1214 added recordWorkflowPhase inside setPhase() without patched() — an ordinary, easy-to-make change. Nothing structural caught it; only a blocking replay gate over real histories would have (and now does).
• Patching is a compatibility bridge, not a deploy-safety mechanism. It cannot stop new code reaching an older in-flight workflow, and a missing patched() is invisible until a long timer fires.
• Nondeterminism errors misdirect: they surface at the next marker check, not at the offending change. The error blamed #1033's letter_mms_in_workflow_v1; the bug was #1214's projection activity. The first diagnosis chased the named marker for two weeks of incident time. Replay the exported history locally at candidate commits — it pins the divergence in minutes and would have falsified the wrong theory immediately.
• Recover by fix-forward, not terminate+restart. The first recovery terminated and re-ran a wedged workflow under a wrong root-cause theory and re-sent a renewal-offer letter to a tenant with a Notice to Vacate on file (unit 414). A wedged workflow is paused, not lost — make the code replay-compatible and it resumes in place with zero side effects.
• Never fork workflow history. The -migrated id space didn't cause the marker mismatch, but it broke signal addressing (F2), tripped the terminal-flag regression (F3), and created a cohort whose existence the team's replay thinking missed.
• Long timers turn a bad deploy into a delayed, silent wedge. Bad code shipped 05-20; damage surfaced 05-27 when a 7-day timer fired. Detection must not depend on a timer firing.
• Alerting that no one wired is no alerting. A week-long customer-facing wedge produced zero pages.
• Migration touched a terminal flag and an addressing assumption and silently broke NTV → decline for real tenants — and that exact bug had been sitting in the backlog (#1027) for a month.

Alternatives considered

• Worker Deployments as the first/only fix — rejected as first move (heavy; doesn't address the synthetic-history root cause; doesn't recover the wedged 3). Adopted later as the structural backstop.
• Patching discipline alone — rejected (it is what failed here).
• Terminate + fresh-start as the primary recovery — rejected after being tried once (2026-06-09): it discards the execution, re-runs every activity, and re-dispatched a renewal-offer letter to an NTV tenant (unit 414) despite the receipt gate (the letter had failed pre-migration, so no receipt suppressed it — receipts only protect already-sent channels). Demoted to last resort behind the no-comms-to-declined guard; see Decision §5 for the fix-forward-first hierarchy.
• temporal workflow reset as the primary recovery — rejected: against replay-incompatible code it replays the surviving prefix and re-wedges, and once the code IS replay-compatible (the fix-forward prerequisite), reset is unnecessary — the workflow resumes in place without rewinding. Retained only as the middle option (--reapply-type None) for histories whose divergence point itself must be discarded.
• Continue-As-New now — promoted from "deferred nicety" to a prerequisite of affordable Pinned; sequenced with the versioning adoption rather than the fundamentals.