0068 — Guarantee turnover work-order creation (and fail loud in AppFolio)

• Status: Proposed
• Date: 2026-06-24
• Deciders: Fede
• Prompting incident: 2026-06-24 TEST-101 call conv_1001kvx867scf5dv4cg0wta2tnk5 — PM walked the unit, approved the recap by SMS, and the turnover stranded at work_scheduled with 0 work orders; nothing reached AppFolio. This is the same class as the 2026-06-09 "all 25 turnovers stuck at 0 WOs" incident and has recurred across dozens of test calls.

Context

The promise we are failing to keep

When a PM finishes a move-out walk and approves the recap, the system must, every single time:

1. Create the work orders (in PropFlow), and
2. Push them to AppFolio — or, if it can't, fail loudly so a human knows.

Today neither is guaranteed. Turnovers repeatedly strand at work_scheduled with zero work orders, and even when WOs are created in PropFlow they can silently fail to reach AppFolio. The failures are quiet: the PM is told the unit is turning while nothing actually happened.

Current architecture (verified from code, 2026-06-24)

The post-approval chain spans four runtimes plus a Temporal workflow:

1. Inbound. PM's "YES" SMS → Twilio → Vercel /api/twilio/webhook. If SQS is enabled and the publish succeeds, the inbound-processor Lambda consumes it; otherwise processEnvelope runs on Vercel via after() (oversize / publish-fail fallback) or inline (SQS disabled). src/app/api/twilio/webhook/route.ts:420-471
2. Confirm. In whichever runtime handled the reply, confirmTurnoverScope runs: it synchronously advances stage → work_scheduled, stamps estimateApprovedAt, writes a tasks_scoped timeline event, and saves. It then fires Temporal signals (signalInspectionCompleted, signalChargeConfirmed). src/lib/domain/turnover/confirm-turnover-scope.ts:235-320
3. Mint (async, Temporal worker). The durable turnoverWorkflow, if it is running and parked at condition(() => state.inspectionCompletedReceived), runs scopeTurnover (mint WOs → PropFlow DDB) → syncTurnoverWorkOrdersActivity (push to AppFolio via the L4 browser agent) → in-house + external dispatch — sequentially, no timers between them. src/lib/temporal/workflows/turnover-workflow.ts:578,630,665,678
4. Inline fallback. confirmTurnoverScope carries a if (!isTemporalConfigured()) { scopeTurnover() } block intended to mint when the runtime can't reach Temporal. src/lib/domain/turnover/confirm-turnover-scope.ts:344-366

scopeTurnover is idempotent — it early-returns {skipped:'already_scoped'} when turnover.tasks.length > 0 — and writes only to PropFlow DDB (it does not call AppFolio). src/lib/temporal/activities/turnover.ts:443-445,575-594

Why it fails — root causes

• RC-1 — The mint depends on a signal reaching a live workflow parked at the exact await. signalInspectionCompleted only does work if a turnoverWorkflow is running and blocked on state.inspectionCompletedReceived. There is no guarantee such a workflow exists for a given turnover (it may predate the workflow flag, never have started, already be completed/terminated, or still be parked on the pre-move-out timer). A signal to a missing/wrong-state workflow silently no-ops. This is the verified TEST-101 failure: confirm ran, the signal fired, no workflow minted, the turnover stranded. (Recovery proved the mint logic itself is healthy — running scopeTurnover directly produced 3 WOs.)

• RC-2 — The inline fallback is dead code in prod. It only fires when isTemporalConfigured() is false. But the inbound-processor Lambda now has TEMPORAL_API_KEY (lambda/inbound-processor/deploy.sh:172,264-266), and Vercel and the Fargate worker are Temporal-configured too. So isTemporalConfigured() is true everywhere in prod → the fallback never runs. The src/lib/temporal/is-configured.ts comment asserting "the agent inbound-processor Lambda is intentionally NOT a Temporal client" is stale and wrong. The 2026-06-24 "root-cause fix" therefore does nothing in production.

• RC-3 — AppFolio sync swallows failures. syncTurnoverWorkOrders "NEVER throws on a sync failure" — a failed L4 write lands the WO at pmsSyncStatus='sync_failed', the activity returns partial, and the workflow logs and moves on. No throw, no alert, no retry sweep. src/lib/maintenance/work-orders/turnover-dispatch.ts:167-170

• RC-4 — No standing reconciler; detection only, no remediation. No cron re-scopes a stranded turnover or re-syncs unsynced WOs. reconcile-turnover-core.ts only advances existing WOs on completion signals and explicitly early-returns when childWorkOrderIds.length === 0. The one mechanism that exists — lambda/metrics-alerts-checker/lib/checks/turnover-stalls.ts — detects work_scheduled + 0 WOs after 15 minutes and fires a warning-severity Sentry alert (engineering-facing, not PM-facing), but takes no remediation.

• RC-5 — Premature success. The PM is told the unit is turning at confirm time, before any WO exists or syncs to AppFolio.

The pattern

Every prior fix patched one runtime or one path. The architecture keeps producing new gaps because no component owns the invariant "approved ⇒ WOs minted ⇒ WOs in AppFolio (or a loud failure)." Reliability is emergent from guessing which runtime handled the reply and hoping a workflow happened to be parked at the right await. That is why it is whack-a-mole.

Lining up with existing patterns (maestro / maintenance vs renewals / tours)

The maintenance ("maestro") WO flow is not a more-resilient model to copy — it has the same three gaps, openly acknowledged in its own code:

• It gates the WO-dispatch signal on isTemporalConfigured() and drops the signal in the Lambda — src/lib/maintenance/work-orders/trigger.ts:61-66: "unlike tours/renewals, maintenance has NO cohort-walker reconciler, so a skipped signal from the Lambda is dropped, not reconciled … the SMS-origin maintenance-signal gap needs a reconciler or Lambda creds (tracked separately)." (= our RC-1/RC-2.)
• Its AppFolio create failure is silent — src/lib/maintenance/work-orders/dispatch.ts:261-273 swallows the error, marks pmsSyncStatus='sync_failed', returns partial, no alert. (= our RC-3.)
• It has no reconciler at all. (= our RC-4.)

One difference favors maintenance and informs D1: maintenance writes the WO row inline (the handler), and only defers AppFolio sync to Temporal — so the row exists immediately. Turnover's row creation (scopeTurnover) itself depends on the signal, making it strictly more fragile. D1 (mint inline at confirm) brings turnover up to maintenance's "row exists immediately" property.

The in-house pattern that does solve dropped-signal reconciliation is the renewals/tours cohort-walker — a Temporal Scheduled workflow (walkRenewalCohortWorkflow → processCohortWalkActivity). D3 forks that proven shape rather than inventing a Vercel cron. (Generalizing D1/D2/D3 to maintenance is out of scope here but should follow — same disease, same cure.)

Decision

Make work-order creation a guaranteed, idempotent, request-time operation, put an always-on reconciler behind it as a safety net, and make AppFolio write failure loud.

D1 — Mint at confirm time, unconditionally and idempotently (remove the runtime gate)

Drop the !isTemporalConfigured() gate. On every approval, in every runtime, confirmTurnoverScope runs the canonical scopeTurnover mint inline. Because scopeTurnover is idempotent (tasks.length > 0 early-return), the Temporal workflow attempting the same mint cannot double-create. The Temporal signal still fires — Temporal remains the owner of the slow downstream (AppFolio sync timing, in-house/external dispatch, the 30-day charges wait) — but the WO mint no longer depends on a signal that can vanish. PM approval and WO creation become one atomic, runtime-independent unit. No code may gate WO creation on isTemporalConfigured().

D2 — AppFolio sync must fail loud

When a WO cannot be created in AppFolio, do not swallow. Reuse the established loud-failure pattern already in this domain (appendTurnoverChargesActivity, src/lib/temporal/activities/turnover.ts:1462-1492): logCritical + emitAlert({ kind:'turnover-stall', severity:'critical', fingerprintScope:'wo-sync-<turnoverId>', runbookUrl }), mark pmsSyncStatus='sync_failed', and surface a truthful non-success state. The WO still exists in PropFlow (nothing is lost), but a failure to reach AppFolio pages engineering immediately instead of sitting silent. (Precedent: the renewal "infra failures → Sentry, not the PM" routing in src/lib/platform/classify-renewal-error.ts and the renewal-letter silent-failure hardening in src/lib/temporal/activities/renewal.ts:1332-1381.)

D3 — Standing reconciler (the safety net) — reuse the cohort-walker pattern

Add a standing sweep that remediates, not just detects. Build it as a Temporal Scheduled workflow, forking the proven renewals/tours pattern (walkRenewalCohortWorkflow → processCohortWalkActivity, src/lib/temporal/workflows/cohort-walker-workflow.ts), not a Vercel cron — so the remediation writes (scopeTurnover, AppFolio sync) get Temporal's retry policy + activity timeouts (not a 60s Vercel function budget) and land on the same Cloud-UI observability surface as the other walkers. New walkTurnoverCohortWorkflow wrapping remediateTurnoverStrandsActivity. It scans for:

• (a) Stranded mint — turnovers at work_scheduled (or otherwise approved) with tasks.length === 0 + a valid projection → re-run scopeTurnover (idempotent).
• (b) Unsynced WOs — WOs on an approved turnover with pmsSyncStatus ∈ {unsynced, sync_failed} → re-run AppFolio sync.
• (c) Escalate — when (a) or (b) cannot be resolved after N attempts, fire a critical emitAlert.

This makes stranding structurally impossible regardless of which runtime approved or whether a workflow was parked. It supersedes the detect-only turnover-stalls.ts check (fold its detection into this remediating walker). It also closes the gap that maintenance has openly tracked but never fixed (see "Lining up" below) — the same walker shape could later cover maintenance's SMS-signal gap.

D4 — Honest PM messaging

Do not claim "the unit is turning / work orders created" until WOs actually exist in PropFlow. If AppFolio sync is pending or failed, the PM sees a truthful state ("work orders created — syncing…" / a clear problem state), never false success. Aligns with the customer-facing-copy bar in CLAUDE.md.

D5 — Real prod end-to-end harness (opt-in merge gate — see the 2026-07-25 revision below)

A harness that exercises the actual runtime routing, not a local function call: place a real call (ElevenLabs simulation against the live agent, or the robocaller fleet) → drive the SMS approval through the real Twilio → SQS → Lambda path → assert WOs appear in PropFlow AND in AppFolio within N seconds. It runs in a prod-like runtime (CI or a deployed endpoint), not a local process calling confirmTurnoverScope directly — that exact gap let the 2026-06-24 fix look green locally while prod was red. Gate it to a dedicated test property; sends suppressed except to owned fleet numbers.

Revised 2026-07-25 — how D5 is triggered. As originally written this said the harness was "wired as a required gate on turnover-touching PRs" via an automatic paths: trigger on opened/synchronize. That trigger is removed. It could never fire the harness: the harness needs a live deploy, preview builds on this repo are opt-in (scripts/vercel-ignore.sh skips every non-main build unless the commit message carries [preview]), and wo-creation-e2e.yml's own paths: filter never overlaps ci.yml's UI-touching preview heuristic — so the automatic path resolved no target and skipped, every time. Evidence (Actions API, read 2026-07-25, .github/workflows/wo-creation-e2e.yml full run history): 67 runs; the harness step executed 14 times — all before the 2026-07-03 targeting fix, all failures — and skipped 53; across the 30 most recent PR runs it skipped 30 of 30. It has never completed successfully from the automatic path, and each run still allocated a runner to reach a foregone skip.

The gate is now opt-in, not automatic: workflow_dispatch, or adding the merge-gate: wo-creation label to a PR (plus a [preview] commit so there is a deploy to target). Nothing that could actually exercise the harness was taken away. It is not a required status check on main and was not one before — branch protection requires Build / Type Check / Unit Tests / review, and the "Require CI" ruleset requires Nested Gate. Whether D5 should become a genuinely enforced gate — which needs a deploy target that exists on every PR, not just [preview] ones — is an open decision, deliberately left to a human rather than assumed here. Details: evals/turnover-walk/wo-creation-e2e-harness.md §5.

No new entities are introduced by this ADR (the reconciler and alerts reuse existing types: Turnover, WorkOrder.pmsSyncStatus, the emitAlert infrastructure). Entity-classification subsection omitted per template.

Consequences

Easier: one invariant, enforced in one place with a reconciler behind it; no more per-runtime whack-a-mole; failures are loud and recoverable; the harness catches regressions against the real system, not a mock of it.

Harder / cost: the WO-mint latency (~seconds) re-enters the confirm reply path in every runtime (accepted — the alternative is no work orders); the reconciler is a new standing job that demands idempotency discipline; the real harness consumes a live call + real AppFolio writes per run (mitigated by scoping to a test property + a merge-gate cadence).

Follow-ups: fix the stale is-configured.ts comment; remove the now-redundant inline !isTemporalConfigured() block (replaced by the unconditional mint in D1); decide whether the Temporal scopeTurnover step formally becomes a confirmation no-op (it already is, via idempotency); upgrade or retire turnover-stalls.ts in favor of the D3 remediating sweep.

Alternatives considered

1. Keep the signal path; just guarantee a workflow is always parked. Rejected — requires guaranteeing workflow lifecycle and await-state for every turnover across deploys and restarts (fragile), and a signal can still race a not-yet-parked workflow. It does not remove the dependency on a deliverable signal.
2. Revert deploy.sh so the Lambda is not a Temporal client, letting the inline fallback fire. Rejected — relies on the runtime gate being correct forever; breaks the moment another runtime handles the reply or creds drift. Same fragility, inverted. D1 is gate-free.
3. Reconciler only, no request-time mint. Rejected — introduces minutes of latency before WOs exist (PM approves, nothing happens for up to N minutes); bad UX and bad for downstream dispatch timing. The reconciler is the safety net, not the primary path.
4. Mint in PropFlow at confirm, sync to AppFolio only via Temporal (no loud-fail/reconciler on sync). Rejected as insufficient — it leaves RC-3 (silent AppFolio failure) in place, which is half the user's pain. D2 + D3 are non-negotiable parts of the decision.
5. D3 as a Vercel cron (initial draft of this ADR). Rejected in favor of a Temporal Scheduled cohort-walker: a cron's remediation writes run on a 60s Vercel function budget with no built-in retry/timeout policy and land off the Temporal observability surface where the other walkers (renewals, tours) already live. Forking walkRenewalCohortWorkflow is the proven, consistent shape.
6. Copy the maestro/maintenance WO flow. Rejected — maintenance carries the same RC-1/RC-2/RC-3 gaps and explicitly lacks a reconciler (its own trigger.ts comment tracks this). It is not a resilient model; the renewals/tours cohort-walker is.