PropFlow.ai PropFlow.ai Docs
Proposed (scoped 2026-07-12)markdown source

ADR-0087 — Per-start namespace routing + a multi-namespace worker fleet

Context

Temporal Cloud has no "move a running workflow" primitive and no "rename a namespace" primitive — namespace names and regions are immutable (ADR-0045, Alternative 3). Every namespace move is therefore always create-new-namespace + migrate-the-population + decommission-old. The only variable is how the population moves.

Today PropFlow resolves the namespace once, globally:

This global-env model is exactly right for ADR-0045's clean cut. At today's scale — ~12 real renewals, one real customer (Camellia), a greenfield target namespace — a stop-the-world cut costs a minutes-to-hours orchestrated window and no throwaway routing code. 0045 correctly rejects a rolling drain for that move as strictly-worse complexity the single-env architecture can't even realize.

But the clean cut is a one-time affordance of being small. Two forces make it un-repeatable:

  1. Stop-the-world stops being safe as the workflow population grows. The 0045 cut works because the live population is tiny and mostly re-materializable (maintenance-comms via signalWithStart, tour via its cohort walker, renewal via a 10-row terminate+recreate). At hundreds of properties and thousands of concurrent workflows across seven domains, a stop-the-world cut means (a) a real inbound-processing outage window that scales with the population, and (b) a terminate+recreate migration whose per-workflow risk multiplies by the row count. There is no version of "pause the whole tenant backend for an hour" that is acceptable once real customers depend on it 24/7.

  2. A single namespace eventually runs out of room. Temporal Cloud namespaces have finite per-namespace throughput (actions/sec), retention windows, and — most importantly — a shared blast radius: a poison workflow, a bad deploy, a runaway signal storm, or a Cloud-side incident on one namespace takes down every domain co-located on it. As PropFlow grows, the tenant backend will need to shard across namespaces — by customer, by region, or by domain-group — for throughput, retention isolation, and blast-radius containment. That shard split is, mechanically, a namespace move — and by the time we need it, force (1) has already made stop-the-world impossible.

So the durable question this ADR answers is: how do we move workflows between namespaces, at any scale, without stopping the world? The answer has to exist before we need it, because the moment we need it is precisely the moment stop-the-world is off the table.

Note also why the "just let old work drain" pattern (the strangler-fig approach 0045's original draft proposed and then rejected) cannot work for this system as currently built, independent of scale:

This ADR proposes the two capabilities that together turn a namespace move into a rolling, reversible, any-scale operation.

Decision

Introduce per-start namespace routing and a multi-namespace worker fleet. Neither is built now; this ADR sanctions the shape so the next namespace move is rolling instead of stop-the-world. Concretely:

1. Namespace becomes a per-startWorkflow decision, resolved from a routing rule

Instead of getTemporalConfig() returning one global namespace, a new resolver — call it resolveNamespaceForStart(routingKey) — maps a routing key to a target namespace at the moment a workflow is started. The routing key is derived from the workflow's natural anchor:

The rule itself is durable config, not a code constant — a NamespaceRoutingRule read-model (see Entity classification) that a start site consults, with a code-level default so an unconfigured environment resolves to today's single namespace (fail-safe: no rule ⇒ current behavior). The critical invariant:

New work routes by the rule at start time; in-flight work finishes where it started. A workflow's namespace is fixed at birth. Signals, queries, and continuation always address the namespace the running execution lives in — never the current rule. This is what makes a move rolling: flipping the rule diverts only future starts.

This requires threading the resolved namespace through every startWorkflow / signalWithStart seam. Today those seams read the namespace implicitly from the cached client; the change makes namespace an explicit input to the start, and makes signal/query helpers resolve the running execution's namespace (from the entity's persisted workflowNamespace stamp — see below) rather than assuming the global one.

2. A multi-namespace worker fleet — every namespace in the routing rule has a live poller

A workflow only makes progress if a worker is polling its namespace. During a rolling move, both the old and the new namespace hold live executions simultaneously, so both must have live pollers. Two viable shapes, decided at build time:

Recommendation: start with (A) for the transient migration window, keep (B) as the end-state for a genuine permanent shard. During a move you want both namespaces polled by one deploy so the transient dual-poller window is a config-list change, not a new ECS service. Once a shard is permanent and load justifies isolation, promote it to its own deployment (B). The routing rule and the worker-fleet config are the two knobs; they move independently.

3. Migration becomes a rolling flip, reversible at any scale

With (1) and (2) in place, a namespace move is:

  1. Stand up the target namespace (register + verify search attributes per-namespace — the ADR-0045 prep-namespace.ts / verify-namespace-prep.ts mechanics generalize), create its schedules paused.
  2. Add the target to the worker fleet so it has a live poller (config-list change under shape A).
  3. Flip the routing rule for the moving cohort (one property / one customer / one domain at a time, or all at once). New starts for that cohort now land on the target; everything already running stays put and drains naturally as those workflows reach their terminal states or re-materialize (on their next signal) in the namespace their entity now routes to.
  4. Decommission the old namespace when empty — leave it read-only through its retention window (histories = audit + rollback forensics), then delete.

Reversible at every step before decommission: flip the rule back and future starts return to the origin namespace; the fleet already polls both, so nothing wedges. There is no stop-the-world window at any scale, because the population never has to move all at once and inbound processing never pauses.

Relationship to ADR-0045 (sequencing, not competition)

Entity classification (per ADR-0027)

Entity Class Naming Spine trace (canonical) OR derived-from / rebuilt-by / drift-tolerance (derived)
NamespaceRoutingRule canonical bare name Durable config — the operator-authored map (routing key → target namespace, + cohort tags + effective-from). No spine anchor (it is infra config, not a human/person entity). Written only by the routing-admin surface; read at every workflow start.
workflowNamespace (a stamped field, not a new entity) — field on the existing per-domain entity (e.g. RenewalSaga.workflowNamespace, WorkOrder.workflowNamespace) Records the namespace a workflow was born into, stamped at start. Signal/query helpers read it to address the running execution. This is the seam that keeps in-flight work addressable independent of the current rule.

NamespaceRoutingRule is the only genuinely new entity; it is canonical infra config with a single writer (the routing-admin surface), no PII, no spine stamp. The workflowNamespace stamp is an additive field on entities that already exist — mirroring how ADR-0085's shard is recomputable but here it must be persisted (a namespace can't be re-derived from the entity once the rule has moved on).

Implementation sketch (rough — this is a proposal, not a plan)

Concrete surfaces a build would touch, so the cost is legible:

The routing-rule config surface.

The start-site change (the bulk of the effort).

The worker-fleet change.

Drift guards you'd want (the same discipline as the rest of the Temporal layer).

Honest cost. This is a large, invasive, cross-cutting change — it touches every Temporal client, every start/signal seam, the worker entrypoint, the infra, and adds a new config surface with its own admin UI and guards. It is emphatically not worth doing until a concrete shard pressure exists; doing it early is the same throwaway-complexity mistake 0045 warns against, just deferred. The value is entirely in when the move comes, it's rolling. Effort estimate: multi-PR, dedicated-session scale (comparable to the ADR-0035 spine migration in seam-count, smaller in data risk since nothing is re-keyed — workflows are born into the right namespace rather than moved).

Consequences

What this commits us to

What gets easier

What gets harder

Follow-up work this implies (when built)

Alternatives considered

  1. Keep the global env + stop-the-world forever (the status quo, extrapolated). Every future namespace move repeats the ADR-0045 clean-cut: pause the tenant backend, terminate+recreate the live population, flip the env, resume. Why it fails at scale: the pause window and the per-workflow migration risk both grow with the population. At 12 renewals it's a minutes-to-hours window and a 10-row terminate+recreate — trivial. At thousands of concurrent workflows across seven domains with real customers depending on 24/7 inbound processing, "pause the whole backend" is an unacceptable outage and the terminate+recreate risk is unbounded. Correct now (which is why 0045 uses it); a dead end later. This ADR exists precisely because the status quo doesn't survive growth.

  2. Strangler-drain on the current architecture (send new work to the new namespace, let old work age out). The pattern ADR-0045's original draft proposed. Why it fails for THIS system: (a) no routing seam — with one global env and every client bound to it, you can't address two namespaces at once, so "new work to new, old work stays" is unreachable without building the very routing layer this ADR proposes; and (b) perpetual domains never drain — maintenance-comms is per-WO signalWithStart/USE_EXISTING and long-lived (maintenance-comms-client.ts), re-materializing on each inbound message and never reaching a terminal state, so "wait for the old namespace to empty" never terminates. Strangler-drain is this ADR — but only once the routing seam and multi-namespace fleet exist. Proposing it without those (as the 0045 draft did) is why 0045 rejected it. In other words: this ADR is the strangler-drain done right; the alternative is strangler-drain attempted on an architecture that can't support it.

  3. A namespace per domain, statically (renewal-ns, maintenance-ns, …). Rejected for the same reason ADR-0045 rejects it (Alternative 2): it breaks direct cross-domain signaling (NTV → turnover → renewal-decline), fragments domains that must coordinate, and multiplies ops — permanently, not just during a move. Per-start routing subsumes the useful part (you can shard a domain-group onto its own namespace when throughput justifies it) without hard-coding the split or severing signaling pairs.

  4. Rename / move the running workflows in place. Impossible — Temporal Cloud namespace names + regions are immutable and there is no native "move running workflow" (ADR-0045 Alternative 3). "Move" always decomposes to create-new + migrate. This ADR is about making the migrate step rolling; it cannot avoid the create-new.

  5. Build this now, ahead of need (skip the 0045 clean cut, do the first move as a rolling migration). Rejected. At current scale the routing layer is throwaway complexity for a move a stop-the-world cut handles in an afternoon — the exact trade ADR-0045 already weighed and rejected. Building the capability before a concrete shard pressure is speculative infrastructure. Correct sequence: cut now (0045), build this when a namespace ceiling or isolation requirement makes the next move real.

Open questions