Turnover System — Architecture Diagnosis & Simplification Plan

2026-08-01 · answering the CTO's four questions: what can we simplify, what can we consolidate or remove, how do we make it deterministic, and what does a nightly end-to-end test look like · evidence: the Monday readiness audit (158 agents, 134 verified findings, 117 distinct defects) and the harness RCA · this doc diagnoses and proposes; it decides nothing
Thesis. Turnover is unreliable for an architectural reason, not a bug-count reason. On 2026-07-31 a live PM demo broke at roughly a dozen hops — while 896 PRs had merged green over the twelve days the Turnovers page was showing zero rows for The Willows. Every hop passes its own tests. What nothing tests is the seams: a voice-config plane that publishes ahead of the app it calls, two recipient records where the code reads one and humans edit the other, a durable workflow that owns the whole second half of a turnover and has never once run, and notification code that reported success unconditionally. Closing 134 findings one at a time does not change that shape. Three structural changes do — one booking engine instead of a scheduler per feature, one gated path for every plane that can change production, and one nightly run that walks the whole chain through the front door and passes only on external artifacts.
896
PRs merged green during the 12 days the Turnovers page showed 0 rows
0
assertions in the nine turnover/vendor eval configs that read state
5
separate representations of one vendor-visit time
5
P0s confirmed — all fixed, deployed and verified 08-01
148 / 42
ADR files on disk vs rows in the index

1. The system today — five planes, five unverified seams

Nothing below is a bad component. Each plane works, is tested, and deploys. The failures live where they meet: two of these planes change production behaviour without passing CI at all, and no test in any lane spans two planes.

VOICE CONFIG push → no CI gate ElevenLabs agents — Triage · Vendor Outbound · Turnover Intake prompts, tool bindings, turn config · one agent is outside the registry entirely 1 RUNTIME CONFIG hand-edited rows Property META propertyEmail · arms · reference mode LEASING_SETTINGS renewalContactEmail / renewalContactPhone 2 CODE CI-gated Vercel app routes · /review · Clara ECS worker Temporal activities Lambda + crons sync · walkers · reconcilers DURABLE flag unset turnoverWorkflow — owns AppFolio sync, in-house dispatch, external dispatch, release, re-list has never started for a Willows turnover · work orders are minted inline, outside it 3 TRANSPORTS the only proof Twilio SMS · voice SendGrid PM + vendor email Outlook / Graph calendar mirror AppFolio work orders · charges 4 5
Boxes are subsystems, not functions. Red markers are the five seams the 07-31 demo broke on.
  1. The voice config plane publishes ahead of the app. sync-specialists.yml is a bare push: branches: [main] while Vercel, Lambda and ECS all gate on CI. On 07-31 at 15:41 MT the live Triage agent — fronting three numbers including Camellia's — had reschedule_vendor_visit bound while production served a build that answered {"error":"Unknown tool"}. The window closed at 16:16 MT; the cause did not. (P1-42)
  2. Two recipient records, and code reads the one humans do not edit. pm-contact-email.ts declares itself the one resolver (ADR-0104 §3.2) and has three call sites; the vendor-outcome, PO-required, forward-to-PM and escalate senders read Property.propertyEmail directly. A fourth resolver exists with different semantics again. One notification splits its two legs across both surfaces — phone from LEASING_SETTINGS, email from META. This bit us again on 07-31: the phone leg was repaired that evening and the email leg still pointed at a @test.local sink.
  3. Nothing owns the slow downstream. ADR-0068 D1 deliberately moved the work-order mint inline at PM approval — "PM approval and WO creation become one atomic, runtime-independent unit" — leaving Temporal the owner of AppFolio sync, in-house and external dispatch, and the charges wait. But TURNOVER_WORKFLOW_ENABLED is unset in both runtimes: prod holds exactly two turnoverWorkflow executions, both RUNNING, zero closed, one parked to 2026-08-21 and one permanently wedged on a patch marker. And on the path that does run, the workflow hits two early phase = 'done' returns before it ever reaches the scope step. The half that was decided is unowned in practice.
  4. Senders reported success unconditionally. sendEmail is total and non-throwing for eight suppression reasons plus hard bounce — one {messageId} return against ten bare return; statements, so a caller cannot tell delivered from suppressed from failed. Three PM notifiers discarded the outcome and returned emailDelivered: true, including the gas-emergency page that tells a tenant "your property manager has been paged." ADR-0119 measured the same class across the SMS side: the dispatch fence covers 7 of 27 dispatch() sites, and email is invisible to it entirely. Fixed and verified 08-01 (#5186, #5178).
  5. Five calendar writers, no shared lifecycle. Create retries three times and writes a DLQ row; update swallows every non-2xx and still reports calendarSynced: true; cancel has no teardown at all on the work-order path. Prod today: 25 Willows work orders hold a live Outlook event id, 22 of them cancelled.

2. Booking should be one engine

The CTO's instruction was to look at how tours do it and standardise on a single way to move events around a calendar. That is the right read: tours already have the spine, and nothing else uses it. Every tour intent — SMS, voice, web, PM edit — enters through applyTourIntent and lands via setTourSchedule, which is guarded twice over: a Symbol capability token that throws at runtime if a caller lacks it, and a CI drift fence that allowlists the five files permitted to import that token, so a new writer fails the build. The schedule row carries an optimistic scheduleVersion; a stale version at timer-fire time skips the effect rather than acting on a moved tour. Slot tokens are HMAC-signed with a 10-minute TTL and fail closed in production — the booking tool has no time parameter to hallucinate.

Vendor visits carry five separate representations of one time (free-form scheduledDate/scheduledTimeSlot, a structured scheduledStartAt/scheduledEndAt pair with a different owner, the call session's own outcome times, the tenant's free-text preference, and the Outlook event itself), written by four unreconciled paths. No write authority, no version, no booking-status model — booked-ness is inferred from Boolean(scheduledDate) — no availability check of any kind, and no CI fence anywhere on the path.

TOURS — one spine VENDOR VISITS — four writers, five representations of one time Tour intents SMS · voice · web · PM edit applyTourIntent — the only door setTourSchedule write-authority token · scheduleVersion optimistic lock availability engine · HMAC slot tokens CI drift fences pin the invariants Tour row · Outlook event · prospect comms gap: the update path swallows failures and still says synced Post-call extraction Tenant slot-pick Reschedule tool / PATCH AppFolio 1-min mirror Work-order row — scheduledDate / scheduledTimeSlot no status model · no availability check · last writer wins Outlook mirror created; rarely updated; never torn down on cancel Tenant notification not on this path — only the handyman-window lane has one
Same problem — put a person and a truck in a room at a time — solved twice, once with a spine and once without.

The proposal is not a new subsystem. It is the tours chokepoint, generalised. One applyEventIntent door that every event type enters — tours today, vendor visits and move-out inspections next, whatever comes after — with one booking-status model, one projection lane and one teardown path. Every property the tours spine already has comes along for free; every gap the calendar findings named gets fixed once instead of five times.

Prospect tour Vendor visit Move-out inspection Future event types applyEventIntent — one door for book · reschedule · cancel write-authority token · optimistic version lock · HMAC slot tokens · availability engine booking status model: proposed → held → booked → cancelled (a cancel is a state, not a deletion) CI drift fences pin every invariant, exactly as they do for tours today Calendar projection idempotent create / update / delete one retry + DLQ policy for all three Notify every party tenant · vendor · PM — one resolver delivery event, not a send attempt Audit + reconciliation one row per state change a sweep that finds orphans and heals CTO rulings encoded in the engine: vendor visits project showAs:'free' so they never block a prospect tour (the engine skips only 'free' — 'tentative' would block) · no same-unit blocking.
Why this is the highest-leverage change. Six of the calendar findings in the audit — phantom visits on cancel, create-once inspections, the silent update failure, the duplicate-mirror race, the fire-and-forget projection, the stranded-mailbox reconnect — are the same defect wearing five costumes: no single object owns the lifecycle of a booked event. One engine collapses them into one fix and one set of drift tests.

3. What to simplify — keep, consolidate, remove

ComponentVerdictWhy
Tours booking spine (applyTourIntentsetTourSchedule)Keep + generaliseThe only scheduler with one door, write authority, a version lock, signed slot tokens, an availability engine and drift fences. It is the template, not a competitor.
Vendor-visit scheduling (post-call, tenant slot-pick, reschedule tool/PATCH, AppFolio mirror)ConsolidateFour writers, five representations of one time, no status model, no availability check. The AppFolio mirror nulls a locally-booked window within 60 seconds; on cancel, 22 of 25 mirrored Willows work orders still hold a live calendar event.
Move-out inspection eventConsolidateCreate-once: a moved move-out date leaves a phantom inspection with no update and no delete path. One prod turnover's two date fields disagree by 27 days with a live event stamped.
Five calendar projection modules, each with its own retry/idempotency/teardown rulesConsolidateCreate retries and DLQs; update swallows every non-2xx and reports calendarSynced:true; the tenant path is fire-and-forget inside a serverless handler. createEvent sends no idempotency key, and the event id is stamped afterwards by a best-effort write whose failure is swallowed — so a lost stamp orphans one event and creates a second.
PM recipient config — Property META vs LEASING_SETTINGS (plus two more resolvers)ConsolidateOne notification splits its phone and email legs across two config surfaces. Humans edit the one the vendor senders do not read.
Two vendor-name matchers (walk path vs dispatch path)ConsolidateThey give opposite answers to the same spoken company name, with inverted safety postures — one auto-assigns at 1.00 confidence while the other reports no such vendor.
Vendor Calling ElevenLabs agent, outside the specialist registryConsolidateNo repo config governs it; a different model at temperature 0.3 where every sibling runs 0.0; no tool a branch adds can ever reach it.
Eight overlapping harnesses + nine promptfoo suitesConsolidateNone reads state in a PR-blocking lane. Replace with one nightly front-door run (§5) plus advisory prose evals.
ElevenLabs sync workflow (bare push trigger)Keep, gate itworkflow_run: [CI] with a success guard. A config plane must never publish ahead of the app that serves it.
/api/simulate/sms as a verification pathRemoveBypasses Twilio, signature validation and SQS while driving real production side effects. Retired as proof, globally.
Vendor arm gatesAlready removed (#5149)CTO ruling: the review queue is the only approval gate. The compensating work is validation inside the approve — shipped 08-01 — not a new gate.
Same-unit tour blockingAlready removed (#5177)CTO ruling: over-engineered. The connected calendar is now the only conflict source.
Turnover durable workflowDecide: own its lanes, or remove itADR-0068 D1 correctly pulled the WO mint out of it; what it was left owning — AppFolio sync, both dispatch lanes, the release, the re-list — it has never run, because the flag is unset in both runtimes. Either arm it and let it own those, or move them to the cohort walker and delete it. A workflow that owns nothing is worse than none: it reads as coverage on an architecture diagram.

4. Determinism — the tiered ladder, applied

Every voice- and text-driven failure on 07-31 sat on the same fault line: a model was asked to decide something a lookup could have decided, or was left to narrate an outcome no code had produced. The ladder is the standing answer — deterministic code first, a strong model only where the ambiguity is genuine, a panel when confidence is low, and a tie fails closed.

1 · Deterministic code answers it normalised ids, roster lookups, explicit precedence, phonetic company fold (#5178), status predicates on every dedupe no model involved 2 · A strong model judges genuine ambiguity one call, structured output, must cite the evidence it used — dispatch intent, scope classification, transcript reads one judgement 3 · Low confidence convenes a panel independent judges, majority rules, disagreement is recorded rather than averaged away N judgements 4 · A tie fails closed refuse, write the reason, page a human — and never narrate an action that did not happen no silent terminal states
Voice-agent authoring rule: an example utterance dominates a prohibition. Two rules banned "Sure" as an opener; two other rules handed Clara "Sure, take your time!" as the model line for a hold. The ban lost on every call. Never demonstrate a phrasing you ban. Corollary from the same fix (#5182): a prompt file that no sync target reads is a written record of a dashboard edit, not a control — the flagged filler survived being flagged because editing the file changed nothing anywhere.
And never let a model narrate work it has no tool to do. The fabricated "Done — Wednesday, August 5" reschedule was not a model failure: the vendor lane shipped with linked_tool_ids: [], so there was no reschedule tool to call. Deterministic rung 4 is what makes this catchable — every completion claim must trace to a tool result.

5. Proving it — one nightly end-to-end run

The CTO asked for nightly tests that walk the whole pipeline including the comms — phone, text, email. The foundation exists: PR #5174 adds scripts/prod-e2e, a single runnable prod harness with numbered legs, a pre-flight that reads every gate before a phone minute is spent, and — the important part — a fidelity ledger: every hop declares itself real or shim, a shim must state what it did not exercise, and a passing assertion on a shimmed hop renders PASS(shim), never bare PASS.

0 Pre-flight identity · suppression A Phone intake PM calls · walk · recap B Text approval signed Twilio webhook C Outbound dial approve in /review D Return call vendor reschedules E Surfaces what the PM sees Evidence a leg must produce to pass real Twilio SIDs inbound AND outbound · SendGrid delivery events (never send attempts) · a calendar event id with a real date · DynamoDB rows at the expected status · the decision record's named fields, not the absence of an error Multi-number personas + a collision guard PM = Fede's cell · vendor persona = Google Voice (number pending) tenant = Twilio harness numbers — lanes must not share a phone Fidelity ledger + Sentry sweep every hop declares real or shim; a shim must say what it skipped the run ends by sweeping Sentry for anything it provoked
real hop — counts as proof shim — renders PASS(shim), never bare PASS pre-flight refusal — costs zero phone minutes
The meta-lesson, from the RCA and from the audit itself. Roll-ups must carry their verification verdicts forward. The original harnesses had zero assertions that read state — 335 javascript, 92 not-contains, 92 llm-rubric, 2 contains-any, and none of them touching DynamoDB, Twilio, SendGrid or a calendar. And this week's audit published two P0s over its own real:false verify verdicts. A summary that drops the verdicts underneath it manufactures confidence in both directions.

6. Standing rulings, and the decision debt

Rulings that are live and enforced in code but written down nowhere central — recorded here so the next branch does not relitigate them:

Decision debt is a live hazard, not paperwork

The corpus can no longer answer "what did we decide?" There are 148 ADR files on disk using 121 distinct numbers22 numbers are duplicated across 49 files, including three-way collisions on 0059, 0060 and 0097 and a four-way on 0101. The README index carries 42 rows, so 106 files are unindexed. Two of the collisions are turnover's own: 0059-vendor-dispatch-for-turnovers.md shares its number with two unrelated ADRs, and 0070-guarantee-move-out-charge-filing.md collides with a conversation-closure ADR. A citation like "see ADR-0059" is now ambiguous on disk — and ADR-0111 makes exactly that citation twice.

Worse than the filing: decisions that never became code, and code that never became a decision.

DecisionStateWhat that costs in production
ADR-0058 — every tool an agent can call really does what it claimsProposed 2026-06-16, taxonomy never builtZero effectClass references exist in source; no generalized drift test. The two tools hand-fixed under ADR-0057 are real, but the generalization that would stop the class recurring is not. The vendor lane then shipped with no reschedule tool at all, and Clara said "Done — Wednesday, August 5."
ADR-0111 — outbound vendor calling agent"Proposed (held for Fede's review — do not merge without his explicit approval)" — and running liveIts business-day approval expiry is not implemented; sessions sat 66 hours before a human bulk-cancelled them. It also carries a 2026-07-31 amendment recording the arm-gate deletion, so the founder ruling lives inside a doc marked un-approved.
ADR-0042 — inbound thread routingProposed since 2026-05-28 — but substantially built and depended on by two later ADRsThe opposite failure: a decision the system already relies on is still labelled a proposal, and its named decision helpers are exercised only by tests while production runs a parallel adapter. Nobody can tell from the corpus that this is settled.
ADR-0104 §3.2 — one PM-recipient resolverDeclared, bypassed (and its number is duplicated)Four vendor and maintenance senders read the property row directly. Seam 2 above.
ADR-0092 — universal suppressionProposed, partially inertIts own gap map names the holes: voice has no consent layer at all, and Microsoft Graph email bypasses sendEmail entirely — which is turnover vendor dispatch's primary leg.
ADR-0068 — guarantee turnover WO creationProposed; D1 shipped, the other half unownedThe inline mint landed. Temporal was left owning "the slow downstream" and is switched off. Seam 3.
ADR-0120 — no in-process side-effect routes as proofNamed, not enforced/api/simulate/sms is its own stated proof case and is still gated by requireUser alone.
The cheap fix is not a re-org. Three mechanical steps make the corpus answerable again: renumber the 27 surplus files, generate the index from the directory instead of hand-maintaining it, and add a CI check that fails on a duplicate number or an ADR referenced by code but still marked Proposed. None of that is a decision — it is the filing system a decision record needs to be worth writing.

7. Where we stand — 2026-08-01


Sources: Monday production-readiness audit — The Willows go-live (amended 2026-08-01) and RCA: why the harnesses missed the turnover happy-path breaks, plus the merged PRs and ADRs cited inline, read from PropFlow-Technologies/propflowai. Live prod numbers were read 2026-07-31 16:30 MT and move; re-read before acting on them. This is a diagnosis and a proposal — nothing here is a decision, and no ADR is superseded by it.
PropFlow Docs