0025 — Adopt Temporal Cloud for renewal workflows

• Status: Accepted
• Date: 2026-05-14
• Deciders: Fede
• File slug: 0025-self-hosted-temporal-for-renewal-workflows.md (filename retained for the link history; the decision is Temporal Cloud, not self-hosted — see Decision below)

Update 2026-05-24 — send-gating contract changed (does not alter the engine/hosting decision below). The env-var send-gating described in Decision item 8 has been retired and replaced. The original TEMPORAL_ACTIVITIES_EFFECT_MODE (shadow/live) + TEMPORAL_LIVE_PROPERTY_ALLOWLIST (per-property allowlist)

• RENEWAL_OUTREACH_KILL_SWITCH (emergency stop) trio is gone. A renewal tenant-send now fires IFF the global RENEWAL_AUTONOMOUS_SENDING === 'armed' (fail-closed — the only arming token is the literal armed, set only via a human-reviewed worker deploy) AND the per-property Property.autonomousRenewalEnabled === true. Ships disarmed by default. The renewal cohort walker (decision engine) is no longer env-gated (RENEWAL_COHORT_WALKER_ENABLED retired) — it is gated solely by its Temporal Schedule's paused/unpaused state. The legacy fireRenewalOutreach path was deleted (one sender). The text below is preserved as the original 2026-05-14 decision record.

Context

The renewal coordination subsystem (RenewalSaga state machine in DynamoDB + four Vercel/Lambda crons + a per-tenant nudge ladder stored as Tenant row fields) has accumulated six weeks of reactive patches — new SLA watchdog states, retry buttons, conflict-error-drop handlers, fix-up scripts, premature event emissions. Every patch is a symptom of the same structural mismatch: DynamoDB has no native timers or signal serialization, so the saga state machine cannot express "wait 24 hours for a tenant reply, then advance to email," and every coordination concern is implemented externally as a poller that may or may not fire.

The trigger incident: 2026-05-14, nine OFFER_PREPARED renewal sagas at Camellia Apartments were silently stuck for 50–333 hours each. Root cause: the renewal-saga-reconciler Vercel cron (the SLA watchdog) had never reached its route handler because src/middleware.ts:341 was 401-ing all cron requests (separate hotfix in flight). With the watchdog silently down, no other system surfaced the stuck sagas. This is the third class-of-bug from the same architectural mismatch this quarter.

A 2026-05-14 architecture review concluded that the underlying problem is "the saga claims to lead but operates as a follower." The fix is to introduce a durable workflow engine that owns the orchestration state (timers, signals, retries, replay) as first-class primitives, and to keep the existing PMS abstraction (PMSClient + PMSWriter + Lease.preparedOffer + activity log) as the read-only mirror of AppFolio.

This ADR locks in the engine choice and the hosting model.

Decision

Adopt Temporal Cloud (managed by Temporal Technologies) as the durable workflow engine for renewal coordination, and for any future PropFlow workflow that needs durable timers / signals / retries.

Specifically:

1. Engine: Temporal (Apache 2.0 OSS, originally built at Uber as Cadence). Workflow code is TypeScript using @temporalio/workflow and @temporalio/activity. Native primitives: await sleep(...), await condition(predicate, timeout), setHandler(signal, fn), queryWorkflow(...), RetryPolicy per activity, deterministic replay, time-travel debugging in the Web UI.

2. Hosting: Temporal Cloud. PropFlow does not run the server side. A pre-apply security audit on a self-hosted alternative found 12 BLOCKING items that would each require a hardening commit and reviewer signoff before any terraform apply — plus the ongoing operational cost of running, monitoring, and securing the cluster ourselves. Cloud trades ~$150–400/mo of additional infra cost for zero ops burden, no security audit gate, no on-call for the cluster itself, and predictable linear pricing. Workflow code is portable between Cloud and self-hosted, so the choice is reversible if needed.

3. Workers: PropFlow runs the workers. Workers are Node.js processes that connect outbound-only over mTLS to the Cloud namespace, poll the propflow-renewal task queue, and execute workflow + activity code. Phase 0 deploys workers on the existing EC2 fleet (alongside the browser agent). If worker CPU saturates that group, workers migrate to a dedicated ECS Fargate service later — that is the only AWS infra we run for Temporal.

4. Region: Cloud namespace in us-east-1 to match the existing Lambda fleet, Vercel iad1, and AppFolio east-coast colocation.

5. Namespaces: propflow-renewal-stage (retention 30 days) and propflow-renewal-prod (retention 90 days).

6. Authentication: mTLS to the Cloud frontend. Client cert/key/CA stored in AWS Secrets Manager (one secret per env, keyed by env). Google SSO for the Cloud Web UI scoped to the propflowai.co domain.

7. Naming: Task queue propflow-renewal. Workflow IDs match the existing RenewalSaga.id format. Activity inputs are IDs only — no PII in activity payloads (PII boundary handled inside activity bodies; see follow-up ADR).

8. Build-on-the-side migration: Workers connect to Cloud from day one with TEMPORAL_ACTIVITIES_EFFECT_MODE=log (shadow mode — no real side effects). Production renewals continue on the saga through Phases 1–3. Phase 4 wires real activity bodies + runs the comparator against real production events. Phase 5 cuts over per-property via a feature flag (flips effect mode to live). Phase 6 deletes the saga.

Consequences

Easier:

• Durable timers (await sleep('24h')) replace the per-state SLA matrix + reconciler cron pattern. State-entered timestamps, watchdog crons, and conflict-error handling on saga writes all disappear.
• Signal handlers (setHandler(...)) replace ad-hoc transition writes on saga callbacks. The Twilio MessageStatus cascade, AppFolio email ingestion, and Clara intent detection all become signal emitters; the workflow runner serializes them deterministically.
• Time-travel debugging: any past renewal can be replayed in the Temporal Web UI, showing every signal, every activity input/output, every state transition.
• Retry policies declared per activity in TypeScript, not as cockatiel wrappers scattered through call sites.
• No new compute surface to operate (Cloud runs the server). No security audit gate. No on-call for the cluster itself.
• Adding a new PropFlow workflow (work orders, prospect onboarding, NTV processing) reuses the same Cloud namespace + worker fleet.

Harder:

• Monthly Cloud cost — ~$0/mo stage (free tier), ~$200–400/mo prod at projected volume. Self-hosted would have been ~$165/mo per env in AWS spend, but with significant engineering + audit overhead.
• Programming model has rules: workflow code must be deterministic, side effects must live in activities, signals must be idempotent. One-day workshop for the team.
• Workflow versioning during deploys requires patched() for in-flight workflows. CI lints for missing patched() calls on non-additive changes.
• Temporal Cloud stores activity inputs/outputs in plaintext in workflow history. The PII boundary moves to the activity body: sanitize at the activity entry, not the log layer. Separate follow-up ADR locks the contract before Phase 4.
• Per-property cutover requires a feature-flag system that does not exist today. Built as Phase 5 prerequisite.
• Vendor dependency on Temporal Cloud uptime + pricing. Mitigated by the fact that workflow code is portable to self-hosted if needed (Apache 2.0 OSS, same SDK).

Follow-up work this implies:

• Temporal Cloud namespace provisioning + mTLS cert flow + worker wiring (Trello card in flight).
• A new ADR for PII handling at the activity boundary (separate, mandatory before Phase 4).
• A new PropertyWorkflowFlag domain entity for the Phase 5 cutover.

Alternatives considered

Self-hosted Temporal on AWS — same SDK, run the server ourselves on ECS Fargate + RDS Postgres + Secrets Manager + KMS + CloudTrail + Config + WAF. Rejected because: (a) the pre-apply security audit identified 12 BLOCKING items in the Terraform module that would each need fixing before terraform apply, (b) ongoing ops burden (Postgres upgrades, ECS task health, mTLS cert rotation, on-call for the cluster) compounds over time, (c) the cost saving (~$150–400/mo) is not material against the security + operational overhead. The Terraform module that was drafted and audited has been removed from the repo; if Cloud ever needs to be replaced, it would be re-authored fresh, not resurrected.

DIY workflow runner — workflow_state table in DDB + cron-driven step advancement + outbox_actions retry table + custom replay tool. Rejected because: at our scale, the durability layer is ≥3 weeks of code we'd write, debug, and own forever. Every team that builds it ships subtle bugs (lost signals, race conditions, non-idempotent retries). Temporal supplies this for free.

AWS Step Functions — AWS-native workflow service, pay-per-step. Rejected because: ASL (Amazon States Language) is a JSON DSL, not TypeScript; tight coupling to AWS service integrations; weaker support for long-lived workflows (14-day waits are unergonomic); workflow code not portable.

Status quo + more reactive patches — keep saga, add more watchdog states, add another reconciler. Rejected: the bug curve is clearly diverging. Three classes of failure this quarter from the same architectural mismatch; each "fix" adds code that the next fix needs to interact with.