RCA: Why the harnesses missed the turnover happy-path breaks

2026-07-31 · ~12 breaks in one live PM demo of the turnover flow · every one shipped through green CI, green promptfoo, and two days of harness building · evidence read from propflowai @ 8a4869d

A. Executive summary. The harnesses were green because every one of them starts after the place where the demo broke: they call tool handlers and model prompts directly, so the transport, identity, suppression, config, and notification hops that failed were never executed. What they assert is also wrong-shaped — the nine turnover and vendor promptfoo suites all run through a single local model provider and grade the words Clara emits, so a "tool fired" in an eval means the model said it, never that a row changed or a message left the building. The two harnesses that do assert real DynamoDB and AppFolio state exist and work, but neither runs on a pull request: one is opt-in by label and has never completed from its automatic path, the other is manual-only. Worse, three of the twelve bugs are actively pinned as correct by passing tests — the router test asserting "vendor routing takes priority over all other routes" is bug 1, and the confirm-stress idempotency scenarios assert exactly the dedupe that swallowed the re-issue in bug 4. The trust you have lost is correctly calibrated: green meant "the model said a sensible thing to a fixture," and we let it read as "a PM can do this."

896

PRs merged green over the 12 days the Turnovers page showed 0 rows for The Willows

eval or harness workflows in main's required checks

turnover assertions anywhere that read DynamoDB, Twilio, SendGrid, or a calendar in a PR-blocking lane

1 / 8

turnover eval domains run per weeknight — each lands roughly every 8 days

10 / 10

most recent PR runs of the one real prod e2e harness: skipped

turnover rows in tracked-gaps.json, the only gap list wired into tooling

B. Per-bug: what should have caught it, and why it structurally could not

#	Bug	Should have caught it	Why it was structurally blind
1	PM-who-is-also-a-vendor eaten by the vendor lane. `inbound-router.ts` L172–198 resolves a VendorMembership and returns `routedTo:'vendor'` before the staff walk at L247–288 ever runs. Message silently lost.	`src/__tests__/inbound-router.test.ts`	The bug is an asserted invariant. L175: `it('vendor routing takes priority over all other routes')` — the test proves the vendor walk wins. And every case mocks `resolveInboundVendorMembership`, so identity is a fixture choice: no test anywhere in the repo constructs one person holding both a PM role and a VendorMembership.
2	Replies to the PM silently suppressed. A 2026-07-21 backfill revoke; his START on 07-22 predated the #4293 reinstate fix (merged 2026-07-22 03:10 UTC) by 54 minutes. Reply lanes pass no `category`, which defaults to `'outreach'` (`twilio/client.ts` L228–237, L285).	`nightly-suppression-canary.yml` → `scripts/canary/suppression-e2e.ts`	The canary supplies the category on every probe (`probe(..., 'outreach' \| 'transactional' \| …)` at L291–346). It asks "does the gate honour a declared category," never "what category does the reply lane actually declare." It also runs against a synthetic identifier on the test property — never a real staff phone carrying a real historical revoke row.
3	Renamed vendor unmatchable. Org reachability is derived from VendorMembership rows (`store.ts` L652–660 / `dynamo/property.ts` L552–566), and the AppFolio sync unconditionally rewrites `company` from PMS on every pass (`pms/writers/vendor.ts` L143–178) — no name-revert guard exists.	`confirm-stress.ts` scenario `vendor-name-resolution`	That scenario runs the real matcher — over a roster the harness itself just wrote. `reseed()` writes `vendors.json` and `vendor-memberships.json` from its own fixture bundle, so membership presence and name spelling are guaranteed by construction. It can never observe an org with no membership row, or a name the PMS reverted an hour ago.
4	Re-issue deduped against a CANCELLED WO. `handle-dispatch-work-order.ts` L196–198 matches on the turnover's task ledger by exact lowercased `"{task} — {vendor}"` label with no status predicate. Returns `ok:true, reused:true`. No new WO, no error.	`confirm-stress.ts` idempotency scenarios	The bug is an asserted invariant. `idempotency-duplicate-confirm` and `-duplicate-add-charge` exist specifically to prove this dedupe fires. The battery has no scenario that cancels a work order and re-dispatches — cancellation isn't in the fixture's state space at all.
5	Approve on /review silently no-ops. The property was left in `vendorJobReferenceMode: 'purchase_order'` by earlier PO testing; `dial-approved-session.ts` L307–338 refuses `po_required`, patches the session back to `awaiting_approval`, and calls `notifyPmPurchaseOrderRequired` — which the PM never received.	`dial-approved-session.test.ts` (8 assertions on this exact branch)	Two separate blindnesses. (a) Config: every test mocks `getProperty` and hands the mode in as a fixture (`h.getProperty.mockResolvedValue({… vendorJobReferenceMode: 'purchase_order' })`), so prod config drift on a live property is untestable by construction. (b) Assertion boundary: the tests assert `{dialed:false, reason:'po_required'}` — the function's return value. Whether a human learns anything is one hop past where the assertion stops.
6	Optimistic-lock race clobbered the approve. `saveVendorCallSession` has a version guard (`store.ts` L1220–1234, `putItemWithVersion` for DDB) that throws `ConflictError`. Its sibling `patchVendorCallSession` (L1237–1244) has none — a blind read-modify-write. It is the function the `po_required` hold uses.	Nothing. No harness runs two writers concurrently.	Concurrency is not in any test's state space: every harness is a single sequential caller. `confirm-stress` is explicitly single-threaded per scenario with a full reseed between; the promptfoo lane has no DB at all. The versioned and unversioned writers sit two functions apart in one file and no test compares them.
7	Turnovers page renders 0 rows for The Willows. The client fetches `/api/turnovers?view=meta` with no `propertyId`; the server strips test-property rows on unscoped reads (`load-turnovers-list.ts` L75–79 → `excludeTestPropertyRows`). Live since PR #4140, 2026-07-19 17:57 UTC.	Any harness that looked at the page a PM looks at	The perfect blind spot. Every turnover harness benches on `appfolio-45` / `isTest:true` — `prove-turnover-e2e-prod.ts` L1148–1155 hard-refuses to run on anything else. The filter #4140 added strips exactly the rows every harness creates, and no harness asserts the UI. The unscoped fetch itself is old (since 2026-03-26); the breakage began the moment the server-side strip met it. 896 green PRs merged over that window.
8	Confirmed booking voided as `wrong_company`. ASR heard "Shabba Cleaners" for "Chapa Cleaners"; `answeringCompanyConflicts` (`postcall-extraction.ts` L159–168) runs `matchVendorCompanyByName` over the ASR string against a roster of exactly one name. Terminal outcome, nothing written.	`promptfooconfig-vendor-call-extraction.yaml` (7 cases, 16 JS asserts)	The eval's assertion boundary ends at the classifier's tool output — it grades whether the model reported the answering company. `answeringCompanyConflicts` is the deterministic consumer downstream of that boundary and is graded by nobody. The corpus is hand-written clean transcripts; ASR noise is not a dimension in it. And nothing lets "we dialed this vendor's own number on file" outweigh a fuzzy string.
9	WO `scheduledDate`/`scheduledTimeSlot` written as literal boolean `true` — calendar mirror had nothing to project.	Any harness reading the WO row after a call	Nothing reads it. The projection eval is a pure module (`turnover-projection/checks.ts`: "no I/O, no env, no imports from src/"); the promptfoo lane has no DB. Honest note: I could not reproduce the mechanism on current main — `normalizeExtraction`'s `str()` guard (L898, since #4616 on 2026-07-26) drops non-strings, and the PATCH route's `normalizeScheduleField` rejects them too. Either it landed via a path I have not traced or the row predates the guard. Needs the actual WO row from the demo — the harness lesson is unchanged either way.
10	No PM notification of any terminal call outcome. Clara said "I'll get that over to Fede"; zero outbound messages.	Nothing asserts an outbound message exists	Three independent silent-drop paths, none covered. (a) Recipients come from prod config — `property.propertyEmail` and `LEASING_SETTINGS.renewalContactPhone`; if both are unset, `notify-pm-outcome.ts` logs and returns with `sends.length === 0`. (b) Both PM notifiers call `sendSms(..., {internalRecipient:true})` with no category — `internalRecipient` skips the frequency cap but not the suppression gate, so this is bug 2 again. (c) All three `notifyPmOfVendorOutcome` call sites are gated on `landed.result === 'written'`; terminal `no_answer`/`failed` sessions have no notify call at all. Every harness asserts return values, never Twilio SIDs.
11	Zero tool calls in the whole reschedule conversation. Clara said "Done — Wednesday, August 5" with nothing bound or invoked.	`promptfooconfig-vendor-calling.yaml` (54 JS + 30 rubric asserts)	Not a model failure — the configuration. `vendor-outbound.config.json` carries `"linked_tool_ids": []` and exactly five platform tools: `end_call`, `transfer_to_number`, `skip_turn`, `voicemail_detection`, `play_keypad_touch_tone`. `vendor-call-context.ts` L9 states it plainly: "No mid-call context tools — the agent talks, transfers, or hangs up, nothing else." No reschedule tool exists in this lane; the write is meant to happen post-call. The eval runs a text provider with no tools at all, so "Done — Wednesday" scores as a clean closing.
12	/review shows no dates — a 2-day-old stale QA approval looks current.	No UI assertion exists anywhere in the turnover suite	Columns are `type, status, name, about, property, details, actions` — no timestamp. The pending cell is the constant string `'Waiting for your OK'` (L410/L422); `capturedAt` renders only inside the drawer (L569). Every harness reads the API or the DB directly; none renders the page, so "the operator cannot tell stale from fresh" is not expressible as an assertion in any suite we have.

C. The systemic patterns

1. Every harness injects mid-pipeline, so the failing hops never execute

confirm-stress.ts imports tool handlers and calls them as functions. It sets DATA_BACKEND=json, delete process.env.DYNAMODB_TABLE_NAME, and leaves Temporal deliberately unconfigured. Even the prod harness cheats the front door: prove-turnover-e2e-prod.ts Link 5 injects the PM's "yes" by POST /api/simulate/sms — and that route calls dispatchInbound in-process, skipping Twilio, signature validation, SQS, and the real conversation-persistence path. Bugs 1, 2, 5, 6, and 10 all live in hops no harness executes.

Evidence: scripts/turnover-replay/confirm-stress.ts L35–47 · src/app/api/simulate/sms/route.ts L136 · scripts/prove-turnover-e2e-prod.ts L654–669

2. Assertions grade model text, not persisted rows or external effects

All nine turnover/vendor promptfoo configs share one provider — file://providers/anthropic-bearer.ts, a direct call to anthropic(). There is no HTTP provider anywhere in the eval suite; no API route is ever exercised. The JS graders parse extractToolCalls(output), i.e. tool_use blocks the model emitted. A tool that "fires" in these evals never executes. No Twilio, SendGrid, DynamoDB, or calendar import exists in any turnover or vendor assert lib.

Evidence: evals/providers/ contains one file · evals/lib/assert-turnover-editing.ts L89–96 · assert-type census across the nine configs: 335 javascript, 92 not-contains, 92 llm-rubric, 2 contains-any — and 0 that read state

3. Single-role, single-org, hand-seeded test actors

Test identity is always a fixture decision, never a lookup. The router tests mock resolveInboundVendorMembership; confirm-stress writes its own vendor-memberships.json; the PO tests mock getProperty. Nowhere does an actor hold two roles. Real customers do: Fede is a PM and a vendor contact, which is precisely bug 1 — and bug 3 is the mirror image, an org relationship that exists in the fixture and not in prod.

Evidence: src/__tests__/inbound-router.test.ts L133–197 · confirm-stress.ts reseed() L119–143 · every vendorJobReferenceMode test reference is a mocked getProperty

4. Prod configuration is never pre-flighted, and some of it is invisible to the repo

Four of the twelve bugs are config, not code: the property's vendorJobReferenceMode left in purchase_order by earlier testing (5), the suppression row from a backfill revoke (2), the vendor roster/membership state (3), and the notification recipients on the property (10). Every harness supplies its own version of all four. This is the same failure shape as the owner-report go-live, which was blocked on unset recipients — a class we have hit before.

Evidence: notify-pm-outcome.ts recipient resolution → sends.length === 0 → logCritical + return

5. Post-call and async pipelines are asserted at the model boundary and nowhere after it

The vendor voice lane is capture-only by design (linked_tool_ids: []): every state change happens after the call, in the extraction classifier and its deterministic consumers. The eval grades the classifier's tool output and stops. answeringCompanyConflicts (bug 8), scheduleFieldsForOutcome (bug 9), and the notify gate (bug 10) are all downstream of that line, and all three are how the demo call ended in nothing.

Evidence: vendor-outbound.config.json · vendor-call-context.ts L9 · promptfooconfig-vendor-call-extraction.yaml asserts only the tool payload

6. Green-and-blind: the harness mints the state it then asserts — and sometimes asserts the bug

A prior session already flagged confirm-stress for this (its own scenario notes admit a WO mint "threw and the checks passed against an empty WO set — asserting nothing," and that the capture gauntlet once "counted 0 forbidden-tool calls — a false pass"). It is worse than an empty assertion when the fixture encodes the defect: the router test asserting vendor-wins is bug 1, and the idempotency scenarios asserting the label dedupe are bug 4. Both go red only when the bug is fixed.

Evidence: confirm-stress.ts L346, L458 · capture-only-gauntlet.ts header · inbound-router.test.ts L175

7. Nothing that could have caught this was allowed to run

This is the multiplier on all six. Main's required checks are Build, Type Check, Unit Tests, review, plus Nested Gate — zero eval or harness workflows, several by explicit written policy. The two harnesses that do assert real DynamoDB and AppFolio state are both opt-in: wo-creation-e2e.yml needs a merge-gate: wo-creation label (10/10 most recent PR runs skipped; its own header records that it "has never completed successfully from the automatic path"), and turnover-pm-sms-harness.yml is workflow_dispatch-only — and it is the harness that found five of these bug classes. The projection runner refuses to start without RUN_LIVE_EVALS=true, with 60 of its 88 scenarios unscored. Nightly rotates one of eight maintenance/turnover domains per weeknight, so each lands roughly every 8 days; vendor-calling, vendor-po-handling, and triage-vendor-po are in no nightly array at all, and triage-vendor-po is excluded from the CI sweep with no catalog floor. turnover-walk-editing-gauntlet passes at a 79% nightly floor. And the one known-gap list wired into tooling (tracked-gaps.json) has zero turnover rows — the list that did name several of these (turnover-intent-coverage.md, 9 of 14 gaps still open, including "change vendor on existing WO" and "vendor not found / dispatch tool error") is prose and gates nothing.

Evidence: require-eval-diff.yml L39–44 · wo-creation-e2e.yml header · nightly.yml L411, L343–352 · run-all.sh L310, L328 · clara-behavior-evals.yml L200–204 (a 25-run sample in which the sweep executed once and 24 green ticks were skips)

D. What changes

The prod e2e harness being built right now in the parallel workflow is the answer to patterns 1, 2, and 5 — it drives the real front door: a real inbound SMS with a real Twilio SID through the real webhook, an approve clicked on /review, a real outbound dial, and assertions against the calendar and the mailbox. That covers the injection-point and the assertion-shape problems. It does not, on its own, cover the other four. Mapping each pattern to its countermeasure:

Pattern	Countermeasure	Concretely
1	Real front door, no in-process injection	The new harness enters via the Twilio webhook with a real SID and signature — never `/api/simulate/sms`, never `dispatchInbound` directly. Retire the simulate route as a verification path: it is fine for poking Clara, useless as proof, and `prove-turnover-e2e-prod.ts` Link 5 should be re-pointed at the webhook.
2	Side-effect assertions are the only green criterion	A step passes when the external artifact exists: a Twilio message SID in the right direction, a calendar event with a real date, an email in the mailbox, a DynamoDB row at the expected status. Model text is evidence in the report, never a pass condition. Where an eval must grade prose, its verdict is advisory and cannot be the last word on a lane.
3	Multi-role actors in the standing cast	Add a bench persona who is both a PM and a vendor contact, and one whose vendor has no membership in the acting org. Those two actors alone would have failed bugs 1 and 3 on the first run. Stop mocking `resolveInboundVendorMembership` in the router suite — resolve against the seeded spine.
4	A pre-flight gate that reads prod config and refuses to start	Before any run: assert the property's `vendorJobReferenceMode` is what the scenario expects, the suppression store is clear for every actor phone/email, the vendor roster and memberships resolve, and notification recipients are set (`propertyEmail` + the outcome SMS number). A missing recipient must fail the pre-flight loudly, not `logCritical` at runtime and return. This gate is also the demo checklist — the demo would not have started.
5	Assert past the model boundary	Grade the deterministic consumers that sit downstream of the classifier — `answeringCompanyConflicts`, `scheduleFieldsForOutcome`, the notify gate — with their own fixtures, including ASR-corrupted company names. Add the identity-precedence rule the code is missing: we dialed the number on file, so a fuzzy name mismatch downgrades to a flag for the PM, never a terminal void.
6	Delete or invert the fixtures that encode the bugs	Rewrite `inbound-router.test.ts` L175 to assert the staff-first precedence, and add a cancel-then-redispatch scenario to `confirm-stress`. Standing rule: a harness may not assert state it minted itself — every assertion reads back through the same door a PM would.
7	Make one thing required, and make green mean something	The new prod e2e harness becomes a required check on turnover- and vendor-touching PRs plus a nightly canary — turnover is the only major lane with no prod canary while renewals, tours, holdover, and suppression all have one. Move the turnover gaps out of prose into `tracked-gaps.json` so the count prints in every sweep. Raise the `turnover-walk-editing-gauntlet` floor off 79%. Put `vendor-calling`, `vendor-po-handling`, and `triage-vendor-po` into the nightly rotation.

And the rule that binds it. No fix from this list is "done" until there is a live prod proof of the user-visible outcome — the PM's phone shows the message, the page shows the row, the calendar shows the visit. Not a passing unit test, not a green eval, not a handler return value. Bug 5 is the argument: the po_required branch had eight passing assertions and still produced a PM who clicked Approve and saw nothing.

E. Honest limits — what even the new harness will not catch

ASR variance. The harness will place a real call, but it cannot enumerate how a given telephony path mangles a proper noun. "Shabba" for "Chapa" is one draw from a distribution. Covered by: making the deterministic consumers noise-tolerant (pattern 5) so a mis-heard name degrades to a PM flag instead of a terminal void — the fix is robustness, not detection. Plus a corpus of ASR-corrupted names harvested from real transcripts, grown every time a live call mangles one.
Live-call UX quality. Whether Clara sounds natural, interrupts well, or lands a close is not assertable. The promptfoo rubrics and the ElevenLabs evaluation criteria are the right instrument for this and should keep running — but only as advisory signal, never as the gate that says a lane works.
Honesty about actions not taken. Bug 11 was Clara narrating an action she had no tool to perform. A harness that asserts side effects catches the absence of the effect, but the fabrication itself needs transcript honesty guards — a check that every completion claim in a reply traces to a tool result, in the same spirit as the existing grounding-check for walk notes.
Config that lives outside the repo. The live ElevenLabs agent's analysis configuration is a dashboard edit covered by no script; nothing in CI can diff it. Until it is in the repo, drift there is undetectable — a small, real supply-chain gap.
Everything nobody thought to script. The twelve bugs were found by a human doing the flow at natural speed with his own phone. That is not a failure of imagination we can automate away. Covered by: keeping the periodic human demo as a scheduled, non-negotiable step — and treating every break it finds as a missing assertion, added the same day.

Read from PropFlow-Technologies/propflowai @ 8a4869d (2026-07-31). Every file path, line number, count, and date above was verified in source or in the GitHub Actions run history; the two places I could not verify a claimed mechanism (bug 9's boolean write, and the exact concurrent writer in bug 6) are marked as such rather than narrated. Bug counts of "12 breaks" come from the demo, not from this analysis.