0045 — Move the tenant backend onto one correctly-named namespace (propflow-prod)

• Status: Accepted — design validated 2026-07-05 (code investigation + independent Fable 5 design review). Execution: Step 1 shipped (PR #3084); prod prep applied + verified 2026-09-02 (30/30); cut procedure corrected 2026-09-02 (#6809, #6814, #6815, #6818, #6822, #6824); Step 5 ref-sweep authored fresh on cut day 2026-09-03 per runbook P4 and merged at runbook Step 4c — which is also the deploy that rolls the permanent renewal-worker service onto propflow-prod (see the runbook's Step-4 preamble).
• Date: 2026-06-02 · Rewritten: 2026-07-05 (corrected scope + clean-cut migration; supersedes the original strangler-drain plan)
• Deciders: Gera (owner), Fede
• Updates: temporal-namespace-strategy.md + src/lib/temporal/README.md (both carry the corrected "current state" on merge — their claim that maintenance already runs on propflow-prod is FALSE; see Finding 0)
• Related: ADR-0025 (adopt Temporal), ADR-0026 (PII boundary), ADR-0044 (NTV → renewal-decline coordination — depends on a shared tenant-backend namespace), ADR-0047 (the read-model that makes restart-from-DDB safe)
• Execution tracker: docs/planning/temporal-namespace-migration.md

Context

propflow-renewal-prod.hmqma is misnamed. It was created for renewals, but it is now the home of the entire tenant backend — not three queues, all of it.

Finding 0 — the load-bearing correction (verified live 2026-07-05)

The original ADR and the strategy doc assumed maintenance already ran on propflow-prod and only renewal/tour/turnover needed moving. That is false. There is no per-domain namespace routing anywhere in the code — every domain resolves a single TEMPORAL_NAMESPACE env var (connection.ts:getTemporalConfig()), and the one Fargate worker (services/renewal-worker/index.ts) opens one connection and registers five task queues on it. So a partial repoint is not merely undesirable — it is impossible without building routing that does not exist.

Live read-only census of account hmqma (2026-07-05):

propflow-prod is greenfield. There is therefore no coexistence / dedup problem in the target — a key input to the decision below.

The problems this creates:

1. Cross-domain signaling is fragile by accident. Workflows in different namespaces cannot signal each other directly. Everything works today only because everything is co-located in the (wrongly-named) namespace. Any future attempt to "clean up" by moving one domain would silently break NTV → turnover → renewal-decline coordination (ADR-0044).
2. Fail-open default (now fixed — Step 1, PR #3084). getTemporalConfig() defaulted TEMPORAL_NAMESPACE to the literal 'default', so a misconfigured runtime silently landed workflows in a 'default' namespace. The resolver now throws on unset.
3. The name lies. Every new engineer reads propflow-renewal-prod and reasonably assumes it is renewal-only; four new workflow domains have silently accreted onto it since this ADR was first written.

Decision

All tenant-backend domains — renewal, tour, turnover, maintenance, maintenance-comms, tenant-confirmation, prospect-outreach, topics, and any future tenant domain — run in ONE correctly-named namespace per environment:

• prod → propflow-prod
• staging → propflow-staging

Why one namespace — one bounded context. These domains are the same tenant-facing backend: same team, same prod data, and they legitimately interact. One bounded context = "same world, different sections." The world is the namespace (propflow-prod); the sections are task queues (propflow-renewal / -tour / -turnover / -maintenance / …), not namespaces. This is the existing split rule.

Operator tooling (Agent Smith, propflow-tools-prod) stays in its own namespace — a genuinely separate trust boundary and runtime. Unchanged.

Migrate with a BOUNDED CLEAN CUT, not an open-ended strangler drain (this reverses the original ADR). The original plan kept both namespaces live for weeks behind a dual-namespace worker while renewal sagas aged out. At the actual scale (12 live renewals, target greenfield) that is strictly worse: the drain is throwaway code (dual-NativeConnection mechanics, per-saga namespace resolution, cross-namespace signal routing — none of it built) whose only "benefit" is avoiding a short orchestrated window we can easily afford. And because one env var drives every domain, the drain's partial state isn't even reachable. So: point all new work at propflow-prod, migrate the small live population in one pass, and decommission the old namespace.

Restart-from-read-model is safe (per-domain), so the cut loses nothing material

Validated 2026-07-05 against the code:

• Renewal — a proven terminate+recreate-from-DDB protocol already exists (scripts/temporal/migrate-sagas-to-workflow.ts); the read-model is retained (ADR-0047). Needs the 4 changes below to target a second namespace.
• Tour — self-heals via tour-cohort-walker (recreate-from-DDB reconciler; timers are absolute-date-anchored to the Tour row → zero elapsed-time loss).
• Maintenance-comms (largest cohort, 26) — entirely signalWithStart-driven with USE_EXISTING; re-materializes in the new namespace on the next inbound message with no migration.
• Turnover — effectively shadow (only live row is a demo); repoint the queue. Re-verify the count on cut day (a real NTV in between creates real state; walkTurnoverCohortWorkflow is the backstop).
• Prospect-outreach (3) — uses raw sleep(); recreate from the inquiry row's outreachTouchIndex or accept the bounded touch-loss.
• Tenant-confirmation (5) — 3-nudges-then-stale, days-scale; recreate from the TenantConfirmation rows or accept.

Accepted caveats (all bounded, none is data loss): relative-timer elapsed time resets to zero on recreate (guarded against re-sends by lease.renewal*SentAt stamps + the receipt store); a live pmStatusOverride: paused is not re-seeded by Phase 0 (re-signal after recreate); and a callback arriving during the cutover window could be missed — the receipt reconstruction + bare-id USE_EXISTING convergence is the backstop.

The migration script needs 4 changes before reuse

It was built to move sagas from a legacy state machine into Temporal within one namespace; a namespace move differs:

1. Add a terminate step with an explicit old-namespace connection — the script currently terminates nothing (its old source was a state machine, so "stops in old, starts in new" is not yet code).
2. Enumerate from live Temporal visibility, not DDB getAllOpenSagas — the DDB list can miss non-saga_* workflowIds (e.g. a live holdover-…-reoffer).
3. Recreate at the BARE renewalId, not renewal-<id>-migrated — the -migrated suffix resurrects a dead addressing shim AND opens a signalWithStart ghost-workflow race; bare-id + WorkflowIdConflictPolicy.USE_EXISTING converges both paths on one execution.
4. Capture + re-signal pmStatusOverride for the ~12 renewals (a for-loop, not a project).

The renewal cohort is Camellia — the sanctioned --confirm-camellia gate (decided 2026-07-05)

Read-only prod scoping (2026-07-05) found the 12 running renewalWorkflows are 10 Camellia (1773625953462, the flagship customer — all real, all OFFER_PREPARED, lease-ends Jul-31 / Aug-31 / Sep-30) + 2 appfolio-45 orphans (eval-reset junk, no saga row). So there are zero non-Camellia real renewals — the renewal cut is Camellia. Options weighed: (A) drain Camellia (wait ~12wk to Sep-30, keep the misnamed namespace + a source worker alive — re-introduces the rejected dual-namespace drain and lingers the wrong name for a quarter) vs (B) migrate Camellia in the cut, gated. Decision: B.

The migration script's Camellia hard-guard is a tooling rail (against blind/accidental writes by ad-hoc scripts — the test bench is appfolio-45, never Camellia), not an absolute prohibition: the app writes Camellia's saga constantly, recover-wedged-renewal.ts is pinned to mutating Camellia's own state, and --allow-camellia did this exact op on 3 Camellia renewals (2026-05-17). So migrating Camellia's own live state is within intent, under gating. The script now:

• Default: abort → skip. A Camellia candidate without the gate lands in skipped as camellia-excluded (loud tally) — it never enters the mutation path; the batch proceeds for everyone else and the 2 orphans skip as saga-missing. (This also fixed a regression where one Camellia row aborted the whole batch.)
• --confirm-camellia=migrate-camellia-renewals (exact literal, fail-closed like --confirm-account) INCLUDES Camellia candidates. Two independent gates: --confirm-camellia gates inclusion (a --dry-run --confirm-camellia previews the full Camellia plan and writes nothing); --apply gates writes. Each included candidate emits a per-candidate stderr WARNING and, under --apply, a pre-op snapshot (saga row + source workflow describe) is captured to the run-log before the first mutation (reversibility evidence). Composes with --saga <id> to name the 10 explicitly. Zero effect on non-Camellia rows.

The 2 appfolio-45 orphans are terminated independently (no saga to migrate) — temporal workflow terminate for each, outside the migration path.

Staging rehearsal must seed an OFFER_PREPARED cohort (the 111 workflows on propflow-renewal-stage are orphan residue with no backing sagas — useless as a rehearsal). Optionally migrate the earliest (Jul-31) Camellia renewal as a canary before the rest.

Migration — execution sequence

Rehearse the whole sequence on staging first (propflow-renewal-stage → propflow-staging; staging has 0 running workflows, so it is a pure mechanics rehearsal).

1. Step 1 — fail-closed TEMPORAL_NAMESPACE (SHIPPED — PR #3084). Removed the 'default' fallback; require the var; throw on unset. Independent + safe.
2. Step 2 — prep propflow-prod (idempotent, days ahead): register + verify the search attributes with a live tagged run (do NOT trust docs — the registry is per-namespace); create the 5 schedules; confirm the prod API key reads the new namespace.
3. Step 3 — fix the migration script (4 changes above) + pre-seed renewal receipts for all candidates (closes the ghost-start re-send window before any traffic moves).
4. Step 4 — cut (minutes–hours; no double-send — see below). Ordering corrected 2026-07-05 — the recreate phase queries each source workflow for pmStatusOverride, which requires a live worker polling the source namespace; and the recreated workflows need a live worker on the target namespace to make progress. So the source worker must stay up through recreate, and a target worker must be up before recreate — the original "deploy target worker first, source goes inert, then recreate" sequence would make every pmStatusOverride query throw and the recreate phase fail mid-cut. Corrected sequence:
   1. Stand up a worker on propflow-prod (target) while keeping the propflow-renewal-prod source worker running — a transient dual-worker window, required for recreate (source worker serves the override query; target worker runs the recreated workflows). Between recreate (4.2) and terminate (4.5) a given renewal exists in both namespaces — the guarantee is no double-send (the pre-seeded receipts + lease.renewal*SentAt stamps make every outreach idempotent across the recreate), not single-namespace residency; the source copy is terminated at 4.5.
   2. Run --phase recreate (source=propflow-renewal-prod, target=propflow-prod, --confirm-camellia=migrate-camellia-renewals — the cohort is 10 Camellia renewals, see above): captures each override from the source, snapshots each Camellia saga pre-mutation, stamps MIGRATED_TO_WORKFLOW + migratedFromState (the Phase-0 resume seed), pre-seeds receipts, and recreates at the bare id on the target with USE_EXISTING. Fully reversible — nothing terminated. The sourceHasPoller preflight fails loud if the source worker isn't live. (Terminate the 2 appfolio-45 orphan workflows independently — no saga to migrate.)
   3. Re-signal any live pmStatusOverride: paused onto the recreated target workflows.
   4. Cut inbound over: flip Vercel + SSM /propflow/prod/TEMPORAL_NAMESPACE + redeploy the Lambdas (mind the Lambda env-REPLACE trap); retire the source worker (target worker stays). New work now lands on propflow-prod.
   5. ⚠️ POINT OF NO RETURN — --phase terminate on old-namespace executions from the visibility list (tour / maintenance-comms / prospect-outreach / tenant-confirmation re-materialize via walkers / signalWithStart or are recreated from their DDB rows).
   6. Pause/delete the old-namespace schedules; unpause the 5 target schedules (created paused in Step 2).
5. Step 5 — sweep hardcoded refs: nightly-renewal-harness.yml, deploy-renewal-worker.yml, scripts/canary/*, run-disarmed-escalate-lifecycle.ts, rehearse-wedge-fix-forward.ts, the 3 Temporal-env Lambdas (inbound-processor, appfolio-sync, metrics-alerts-checker — it monitors Temporal reachability and will page during the cut), README.md table, TEMPORAL_INFRASTRUCTURE.md, the strategy doc, this ADR's status.
6. Step 6 — decommission: leave propflow-renewal-prod read-only through its 30-day retention window (histories = audit trail + rollback forensics); delete after 30 quiet days.

Point of no return = terminating old-namespace executions (Step 4.5). Everything before it is reversible by flipping env vars back. Mid-batch, the script is per-saga idempotent via a run-log (--saga <id> retry; each of stamp / recreate / re-signal is a resumable run-log flag — NOT gated on saga state, since the migration itself sets MIGRATED_TO_WORKFLOW and a state-gated skip would strand a stamped-but-not-recreated candidate after a crash); a failed saga leaves its old execution intact (nothing terminated yet). Worst case: flip all env back and terminate the new-namespace recreates — symmetric, because the protocol is namespace-blind.

Consequences

• Unblocks turnover ↔ renewal ↔ maintenance signaling (ADR-0044) — all in one namespace.
• Fail-closed namespace — a misconfigured env hard-fails at boot instead of silently using default (shipped, PR #3084).
• Correctly-named foundation while the company is small — the cheapest this migration will ever be; every month of delay accretes more live state and more code pinned to the wrong name.
• Cost: a short orchestrated cut window (minutes–hours) + the 4 script changes + search-attribute re-registration. No dual-namespace worker, no weeks-long drain, no cross-namespace signal-routing shim.

Alternatives considered

1. Original strangler drain (superseded). Dual-namespace worker + per-saga namespace resolution + event/DB-path cross-namespace routing, kept live for weeks. Rejected: throwaway complexity that the single-env-var architecture can't even partially realize, to avoid a window we can afford at this scale.
2. A namespace per domain (rejected). Breaks direct cross-domain signaling; fragments domains that must coordinate; multiplies ops.
3. Rename in place (impossible). Cloud namespace names + regions are immutable; there is no native "move running workflow." "Rename" always = create-new + migrate.
4. Leave as-is (rejected). Keeps the lying name and the accidental-colocation fragility; guarantees more domains silently accrete onto the wrong namespace.

Open questions

• O1 — CLOSED. Maintenance's resolved prod namespace confirmed = propflow-renewal-prod (Finding 0), NOT propflow-prod.
• O2 — CLOSED (approach). Search-attr registration + verification is scripted (prep-namespace.ts + verify-namespace-prep.ts); the verifier uses a per-attribute visibility query (Cloud refuses operatorService.listSearchAttributes with a namespace-scoped key), which IS the "verify via a live tagged run" O2 asked for. The exact attribute list is derived from the upsertSearchAttributes call sites, not docs. Live registration on propflow-prod remains a human-gated apply (Phase C).
• O3 — CLOSED. Prospect-outreach: accept the bounded touch-loss (3 live workflows, raw sleep(); recreate-from-outreachTouchIndex is throwaway complexity for an at-most-one-nudge effect already guarded by lead state). They terminate on the source and re-materialize from their inquiry rows.
• O4 — Staging rehearsal date + cut-day window (Gera to schedule).