Willows Funnel — Pass/Fail
Autonomous end-to-end test of the leasing funnel on the Willows test property, synthetic prospect, no signature. Scope (Fede): approved → contract out → questions → change terms. Signature is out of scope — it tests AppFolio's e-sign, not our system; Fede clicks it manually once this funnel is 100% green.
PASS works end-to-end, verified with evidence
FAIL broken, cause identified
PARTIAL works with a gap
VERIFYING test in progress
Stage 1 — Approved → Contract Out
| # | Use case | Status | Evidence / break point |
|---|---|---|---|
| 1 | A real AppFolio rental application can be created for a prospect on a vacant test unit | PASS (live) | Application 42 — "Willow T. SyntheticBCEIGH", unit 356 — created live on the test property (Aug 24, ~2:45am MT). First real rental application this property has ever had. The two field names the prior session had to guess were resolved by READING the form instead of firing at it (scripts/capture-application-fields.ts dumps all 186 input names + which are aria-required): DOB is contact_info[birthdate] and SSN is contact_info[tax_id] — the guessed applicant[date_of_birth]/applicant[ssn] do not exist on the form at all, which is why they never produced an error to learn from. rental_application[desired_move_in] was also aria-required and missing. Two further fail-closed rounds caught content rules (last name rejects digits; tax_id rejects "groups of 0s"). Pushed to #291. |
| 2 | Application can be moved to Approved in AppFolio (manual, no screening required) | PASS (live) | Application 42 is Approved, verified by re-reading the page (not by trusting a POST status). The HTTP-replicated update had been returning 500; a click-capture showed why — the real request is POST /rental_applications/42?block_name=application_status with _method=put and rental_application[status]=approved (lowercase), plus the approval-conditions family the form always submits. The module was sending _method=patch, status=Approved, and no block_name. Gotcha for anyone automating this page: the status block is #application_status_show with its OWN a.js-edit-block; the first generic "Edit" belongs to the applicant name/contact block. |
| 3 | PropFlow sync ingests the approval → prospect stage becomes APPROVED | BLOCKED — units lack AppFolio pointers | The sync is healthy and DOES see the application: property 45's rental-application feed went from rows=0 to rows=1 minutes after creation, and the job runs every 5 minutes (propflow-appfolio-sync-prod-rental-applications-schedule, ENABLED). But every cycle ends: property=appfolio-45 rows=1 created=0 updated=0 unmatchedUnits=1 → "1 applications had no matching PF unit — backfill unit pointers". Exact cause (verified in prod DynamoDB): the writer indexes units by getAppfolioUnitExternalId(u) => u.af?.unitId (pms-pointers.ts:4, used at rental-application.ts:608). Property 45 HAS its 33 unit rows — including UNIT#EVAL-MI-33041, which is AppFolio unit 356 — but those rows carry no af block at all, so af.unitId is undefined, the index is empty, and AppFolio's numeric unit id matches nothing. The writer then skips rather than create an orphan prospect (its own comment: "this surfaces a missing unit pointer that the property's onboarding sync should have stamped"). Latent gap worth remembering: the test property's units have never carried AppFolio pointers, so ANY application-sync test on property 45 would have failed this same way. Rows 4 and 6 depend on the prospect existing, so they inherit this block. The fix, and why it wasn't run tonight: the standard path exists — scripts/appfolio-annotate-ids.ts ("Annotate AF IDs onto PropFlow records (Unit.af.unitId, Tenant.af.tenantId, Tenant.af.occupancyId) … the prerequisite for the sync processors to resolve rows"), dry-run by default, whose documented example is Camellia. It matches AppFolio rent_roll rows to PF units by unit NUMBER and writes only the af block. It was NOT run against prod overnight for two reasons: the local env points at propflow-dev, so running it on production means hand-overriding the table target; and it also stamps TENANT pointers on a property carrying live tenant occupancies and 27 renewal sagas — a blast radius past the thing being fixed. Morning ask for Fede: bless running appfolio-annotate-ids.ts against prod for appfolio-45 / AF 45 to stamp its unit + tenant pointers. |
| 4 | Reaching APPROVED creates a review item (approval hook) | BLOCKED BY ROW 3 | The hook itself is WIRED and merged (PR #6172, Aug 23): sync registers the approval trigger → review-item creator, keyed off the real synced record, landing dark (fires only for armed appfolio-45). It was dead code (zero callers) before that. It still cannot be re-tested, because it fires off the PROSPECT the sync creates — and the sync currently skips the application for want of unit pointers (row 3). A real approved application now exists in AppFolio (app 42), so the moment row 3's pointer backfill is blessed, this hook gets its first real exercise on the next 5-minute cycle. |
| 5 | Review item shows the real recipient (not blind approve) | PASS | Live-verified: the review drawer reads "GOES OUT AS: Email — fedechagu@gmail.com"; approve goes inert when empty (#6162, deployed). |
| 6 | Congratulations message approved → sent to the correct channel/recipient | CODE-VERIFIED · NOT LIVE-PROVEN | Honest status (Aug 24): the approve path is correct by inspection — the decision route sends the EXACT draft the reviewer saw (congratsMessageDraft) on the EXACT channel they approved (item.channel: sms → SMS, email → email), and deliberately does NOT silently re-route to the other channel if the stored one turns out undeliverable at send time (that would send a message on a channel the audit row does not record). Row 5 separately live-verified that the review item shows the real recipient and goes inert when empty. What is MISSING is a live end-to-end proof: no real approved applicant has flowed through this path yet (the test property has never had a real application — see rows 1-3), so nothing has actually been delivered to a recipient by this route. It is marked PARTIAL rather than PASS for exactly that reason. |
| 7 | Lease prepared with real terms (rent/dates), blank-money guard fail-closed | PASS (live) | The e-sign chain hydrated the 15 merge fields via browser-fetch and CREATED the lease document on occ 1047 with real terms — the blank-money/blank-contract guard passed (it correctly blocked the prior empty-hydration attempt). Document exists in AppFolio. |
| 8 | Lease sent out for signature (contract out) — happy path, to the prospect | SEND PROVEN LIVE · CODE PORT REMAINING | CORRECTION (Aug 24 eve): the "no online template" conclusion further down was WRONG. A live capture proved the full send end-to-end on the test property (synthetic tenant, on-screen confirmation): the "Camellia Lease" template IS present and pre-selected in the "Prepare Lease" second tab — earlier automation was accidentally de-selecting it, and the "No lease templates" banner belonged to a different, unused feature. Remaining work is pure code: port the captured send chain into the module, then a real-terms run via the already-built cancel→regenerate (the proof lease carried $0 defaulted rent). Superseded diagnosis kept below for history. Overnight (Aug 24): the fresh no-hands chain now runs clean end-to-end on a freed test unit — login→pick vacant unit 360→profile→select units→lease details→template resolve→merge-field hydration (now correctly browser-routed, was 401'ing on raw Node)→lease document CREATED with real terms (doc 6462644). It fails ONLY on the very last hop: the send hop — after POST /leases/documents_app_proxy/documents creates the doc (6462644), the send PUT /leases/documents_app_proxy/documents/6462644/send_all_to_tportal returns HTTP 404. #290's inferred document-scoped send endpoint is wrong (renewals send via /renewal_offers/{id}/, applications via /rental_applications/{id}/ — an app_proxy doc with no application has neither). Raw-CDP investigation (Aug 24) strongly indicates the app_proxy CREATE already SENDS: driving occ 1050's live flow, generate_lease/edit redirects FORWARD to finish_move_in (you cannot go back — the lease is already generated), and the flow sits at finish_move_in with an empty "Lease Signed Date" — the exact awaiting-signature state occ 1047's confirmed out-for-signing lease sat in. So the module's separate send PUT is very likely redundant. Two concrete fixes for the code path: (a) if a send IS needed, use the create response's data.meta.lease_document_id (the module's OWN documented fallback id) rather than the app_proxy data.id that 404s; (b) otherwise drop the redundant send. Everything up to document creation is proven no-hands; the contract-out hop is confirmed-by-flow-state, pending a direct tenant-portal/status read (occupancy include APIs 500 on these fresh occupancies, so not yet read out-of-band). UPDATE (11:24pm): read the app_proxy CREATE response directly (doc 6462660) — it returns a BARE document (type "documents", resource_type "lease", empty signing_data/parties/templates relationships) with links {metadata_url:/documents/{id}, file:/documents/{id}/file} — NO send link and NO data.meta.lease_document_id (fallback id is undefined). So the create response gives no send affordance, the doc lives at /documents/{id} (not the app_proxy path the send targets), and both guessed send endpoints (/leases/documents_app_proxy/documents/{id}/ and the fallback) 404. Next: capture AppFolio's OWN generate->send from the UI with full GET+POST to read the real endpoint. (Also confirmed: the raw-CDP cancel of occ 1050 completed — unit 360 came back free and was re-picked.) |
| 9 | Signing link persisted + retrievable later ("where's my lease?") across channels | RETRIEVAL BUILT (#6199 MERGED) · TOOL WIRING PENDING | Prospect page shows it live-verified ("Lease out for signing · Signing link sent Aug 23", #6166 deployed). The cross-channel gap turned out to be a missing KEY, not a storage location: getLeaseSigningStatus takes a prospectId, but an inbound voice/SMS "where's my lease?" only knows WHO is asking (a personId from the phone/email). #6199 (merged Aug 24) adds getLeaseSigningStatusByPerson(personId, propertyId?) — resolves person → prospect → signing state via the same getProspectByPersonId the change-request lane already uses, scoped by property. Degrades identically to the prospect path (unknown person / no prospect row / throwing lookup / no portal URL → found:false, plain-English message); never fabricates a link, never sends. 9/9 tests. Remaining: registering the Clara tool entry that CALLS it — that touches the external agent runner and needs its own handler + evals, so it was deliberately left out of an overnight change to a live conversational surface. |
Stage 2 — Questions → Change Terms
| # | Use case | Status | Evidence / break point |
|---|---|---|---|
| 10a | Prospect asks a lease question on a normal thread → Clara answers from known state | PASS | Verified in conv 669d3c40: service animals, occupancy, payment-method all answered correctly (lease-answer-context.ts). |
| 10b | Lease question arriving while the thread is already escalated → answered, not held silent | FIX MERGED · AWAITING LIVE RE-TEST | Fede ruled (2026-08-23): "always be helpful with the questions we can answer; escalated threads are separated from questions we can answer" — an escalation holds ONLY its own topic. The fix (#6175) was reviewer-blocked most of the night and has now MERGED (Aug 24, 3:45am UTC) after four review rounds: contentless acknowledgements can no longer release the hold, the acceptance harness was taught to see risk rather than being blind to the change, and it was validated by a 30-day replay of 670 real resident messages with the affected rows human-labeled. Remaining: a LIVE re-test — a real lease question arriving on an escalated thread and getting answered. That can piggyback on the row 3→6 chain once the unit-pointer backfill lands and a synthetic applicant is flowing. |
| 11 | Prospect requests a term change (e.g. move-in date) → lands as a review item, not a free-text PM email | MERGED + DEPLOYED · TESTABLE | Built + merged (#6174, deployed to prod Aug 23 ~9:42pm MT). When an approved applicant whose lease is already out for signing sends a change-of-terms ask, Clara's forward-to-PM handler now captures a structured NewLeaseChangeReview (requested field + verbatim text) instead of a free-text forward. Trigger needs a field word + a change word (a plain "what's my move-in date?" intentionally does NOT fire). Test: as an out-for-signing test prospect, text Clara "Can I push my move-in date to October 15 instead?" → it appears as a structured item in the review queue at propflowai.co/review. (Approving it = stage 12, still fail-closed — see below.) |
| 12 | Approving the change → lease is canceled + regenerated with new terms | CANCEL PROVEN THROUGH MODULE · #292 MERGED | Design locked from a live read-only capture. AppFolio cancels a sent-not-countersigned lease via the "Cancel Move In" action = POST /move_in_flow/finish_move_in/cancel_flow (AppFolio KB "Cancel a Tenant Move In", Scenario 2, unit-page path: "the related lease will be canceled and removed from your database") — not a separate cancel-lease screen. Because our move-in has NO bound rental_application, canceling wipes the whole move-in with nothing to reuse, so regenerate must recreate from scratch: runCancelMoveInL4 → runNewLeaseE2eL4(new terms). Modules built code-side + tested (runCancelMoveInL4, 29 tests, PR #292 fail-closed, scope-guarded, HARD_BLOCKs a countersigned lease). CANCEL HALF PROVEN LIVE (Aug 24): using raw Playwright over CDP (chromium.connectOverCDP on the Browserbase session — the Stagehand wrapper couldn't do clicks/capture), the click-through executed a real Cancel Move In on occ 1047: "Continue Move In" REGENERATED a live flow (fresh web_flow_id 1618 — AppFolio's move-in web_flows are ephemeral, which is why stored ids go stale), clicked Cancel Move In → confirm → redirect to dashboard. Verified: occ 1047 no longer appears in the units listing, and the freed unit 360 was immediately picked up by the next fresh create→send run. So the cancel WORKS and frees the unit. Remaining for a full 12→13: (a) the exact cancel request shape for the code path (runCancelMoveInL4's direct POST 404s — being re-captured WITH GET included, since the UI's real verb differs); (b) the regenerate+resend depends on stage 8's send hop, which currently 404s (see row 8). Cancel proven; regenerate/resend gated on the send-endpoint capture. UPDATE (11:39pm): captured the REAL cancel request via raw-CDP with GET included — it is GET /move_in_flow/finish_move_in/cancel_flow?web_flow_id=… → 302 → dashboard (a GET with web_flow_id as a query param; a POST 404s). Fixed runCancelMoveInL4 to fire exactly that, tests 29/29, and VERIFIED IT LIVE THROUGH THE MODULE: on occ 1043, raw-CDP Continue Move In regenerated a live web_flow, then runCancelMoveInL4 returned status 200 and canceled it. #292 MERGED (sha 2b19e2f). Stage 12 cancel is done end-to-end through the code path; only the regenerate→resend half (stage 13) remains, gated on stage 8's send endpoint. |
| 13 | Regenerated lease is resent (contract back out) with the new terms | BLOCKED BY ROW 8 | Depends entirely on the send hop. The cancel half of the change-terms loop is PROVEN live through the module (row 12), and the regenerate path is built (runCancelAndRegenerateNewLeaseL4 chains cancel → full create+send), but "resent" cannot happen until a lease can be SENT at all — which is blocked on the account having no online lease template (row 8). Once that exists and the real send request is captured, this row is exercised by the same run that closes row 8. |
Cross-cutting guardrails
| # | Use case | Status | Evidence / break point |
|---|---|---|---|
| 14 | No write ever lands on Camellia or the wrong property (scope guard fail-closed) | PASS | Universal scope guard merged (#287); validated live when it blocked a finish-move-in POST |
| 15 | No fabricated facts / no promises without dispatch in any message sent | PARTIAL — GUARDS IN PLACE, NO E2E SWEEP | Honest status (Aug 24): real guards exist and some are proven — the blank-money/blank-contract guard fail-closed live (it blocked an empty-hydration lease attempt, row 7); property data was cleaned of fabricated codes/fees and verified live; the voice fabrication fix is in lab. But there has been NO end-to-end sweep of the messages this funnel actually produces (congrats + change-request replies) asserting zero fabricated facts and no promise-without-dispatch — and since no message has been delivered end-to-end yet (row 6), there is nothing delivered to audit. Cannot be called PASS until the funnel produces real messages and those are swept. |
| 16 | Lease-finalize / countersign / execute never fires autonomously | PASS | HARD_BLOCK tier refuses fully-executed + finish_move_in + countersign; proven blocking live |
Status as of 2026-08-23, updated live as the two autonomous verification lanes report. Synthetic prospect throughout; zero real-person contact; property 45 only. Signature (AppFolio e-sign) is deliberately out of scope — Fede's single manual step at the end.
Decision for Fede: online lease templates, or Clara cannot send leases
Stage 8 ("lease sent out for signature") is blocked, and overnight investigation narrowed it to a single decision only Fede can make. Both branches are mapped with evidence.
| What was checked | What was found (evidence) |
|---|---|
| Does the account have online lease templates? | No. /lease_templates holds exactly one row — AppFolio's built-in "Sample Lease Template" — and the move-in wizard's online option states verbatim: "No lease templates have been set up." Templates are account-wide (not per-property), so this is true for Camellia as well as the test property. |
| Could the wizard's default offline (PDF) path serve as a fallback? | No. Clicking its "Prepare Lease" button fired zero server requests, produced zero tenant-visible signals (no portal, no e-sign, no "sent" language), and simply hands off to /leases/form_templates/documents/new — an in-app PDF document builder. It has no delivery step at all: a PM prepares a PDF and handles signing outside the system. |
| So what does that mean? | The offline path is not a half-built stage 8 — it is a different, manual workflow. Automated contract-out requires an online lease template. There is no engineering path around it. |
Why the send machinery itself is not in doubt: the RENEWAL e-sign path already sends leases for signature in production — and it does so by driving AppFolio's own UI (it opens the renewal document form, clicks "Prepare Renewal" then "Send Renewal for Signatures", and captures the resulting documents POST + send_all_to_tportal PUT that the page's own JS fires). Its code comments record why: replaying those calls from outside the page kept 403'ing on /forms/field_value_sets, and "8 deploys of header/CSRF/cookie tweaks didn't fix it" — letting the UI click means the form's framework supplies the values, headers and CSRF correctly. So send_all_to_tportal is real and proven; the new-lease module's error was trying to HTTP-replicate that endpoint instead of click-capturing it the way renewals do. That strengthens option (a): once a template exists, the fix is to mirror the renewal's click-capture approach for the move-in flow — a known, already-working pattern in this codebase, not new invention. This also means the send PUT should NOT be deleted as redundant.
The decision: either (a) create an online lease template (AppFolio → Leasing → Lease Templates → new from sample; account-wide, a real content-authoring task), after which the automated generate→send lights up and the remaining send endpoint can be captured in one run — or (b) accept that Clara cannot send leases for signature, and stage 8/13 get redefined around a manual step. This was deliberately NOT done autonomously: a lease template defines the actual contract Clara puts in front of a resident.
Everything downstream is ready for (a): the create/generate chain is proven no-hands with real terms, cancel is proven live through the module, and the capture harness is one command from reading the real send endpoint (see the raw-CDP runbook below).
Pre-staged unblocks — ready to run on Fede's word
Both overnight blockers are configuration/data, not code. Each is reduced to one command, staged so there is nothing left to figure out.
| Blocker | What to run |
|---|---|
1. Unit/tenant AppFolio pointers missing on the test property — blocks rows 3, 4, 6. The application sync sees the application every 5 minutes and skips it (unmatchedUnits=1) because property 45's unit rows carry no af.unitId. | Run in ~/.claude/propflowai/. Dry run FIRST (the script is dry-run by default — it prints the exact diff before writing):DYNAMODB_TABLE_NAME=propflow-prod npx tsx --env-file=.env.local scripts/appfolio-annotate-ids.ts appfolio-45 45 --user-id <pmscred-user>Review the printed diff — it should stamp af.unitId on ~33 units, and EVAL-MI-33041 must come out as 356. Only then re-run with --apply appended. Two cautions: the table override is required because the local env points at propflow-dev (so an un-overridden run silently does nothing to prod); and the script also stamps tenant pointers on a property that carries live tenant occupancies and 27 renewal sagas — worth a glance at the tenant half of the diff before applying. After it applies, the next 5-minute cycle should land the prospect (row 3), which fires the approval hook (row 4) and enables the congrats delivery test (row 6). |
| 2. No online lease template on the account — blocks rows 8 and 13. Only AppFolio's built-in "Sample Lease Template" exists, account-wide. | First Fede decides online-vs-PDF (see the decision section above). If online: create the template in AppFolio → Leasing → Lease Templates → new from sample. Then, in ~/.claude/appfolio-browser-agent-new-lease-template-resolve/:CAPTURE_UNIT=348 CAPTURE_TEMPLATE="Camellia Lease" npx tsx scripts/cdp-capture-move-in-send.tsThat drives AppFolio's own UI (the renewal path's proven click-capture pattern) and writes the REAL send request to /tmp/move-in-send-capture.json — which is what the new-lease module gets pointed at, closing stage 8 and unblocking 13. |
Raw-CDP capture tooling (how tonight's evidence was produced)
The Stagehand wrapper cannot do real clicks or network capture (page.act / page.on('request') both throw). Everything below uses raw Playwright attached to the SAME Browserbase session via chromium.connectOverCDP(session.connectUrl) — that is what cracked the cancel and diagnosed the send. Scripts live in the worktree ~/.claude/appfolio-browser-agent-new-lease-template-resolve/scripts/ and are intentionally NOT committed (they import playwright-core, which is only a transitive dep — CI typecheck cannot resolve it, and adding a dependency for ad-hoc capture tools is not worth it).
| Script | What it does | How to run |
|---|---|---|
cdp-cancel-flow-1047.ts | Proven cancel click-through. Occupancy page → "Continue Move In" (regenerates a LIVE web_flow — AppFolio's move-in flows are ephemeral, stored ids go stale) → "Cancel Move In" → confirm, capturing requests. This produced the real cancel request: GET /move_in_flow/finish_move_in/cancel_flow?web_flow_id=… → 302, which fixed runCancelMoveInL4 (#292). | set -a; source .env.local; set +a; npx tsx scripts/cdp-cancel-flow-1047.ts — edit const OCC to target a different occupancy. |
cdp-capture-generate.ts | Staged send-capture harness. Unit → occupancy → Continue Move In → generate_lease step → select the online ("Lease Templates") radio + template → click the generate/send action (never Skip / Upload-and-Mark-Fully-Executed / Done), capturing every GET+POST. This is the one command to run the moment an online lease template exists — it yields the REAL send endpoint. | npx tsx scripts/cdp-capture-generate.ts — edit const UNIT. Note: the template picker is a custom JS widget, not a native <select>, so selectOption silently no-ops; that step needs a click-open-then-click-option once a real template exists. |
cdp-capture-move-in-send.ts | THE one to run once an online lease template exists. Mirrors the RENEWAL path's proven click-capture (apiClient.ts: apiPrepareRenewalAndSendViaClick) for the move-in flow: navigate to generate_lease → attach the capture → select the online radio + template (handles both a native <select> and a custom widget) → wait for React hydration → click prepare, then send — capturing the documents POST and the real send_all_to_tportal request. Carries the renewal's two hard-won lessons verbatim: attach the observer AFTER navigation (navigating wipes it), and never click before hydration (an unhydrated POST body sends the tenant a BLANK lease). | CAPTURE_UNIT=348 CAPTURE_TEMPLATE="Camellia Lease" npx tsx scripts/cdp-capture-move-in-send.ts → writes /tmp/move-in-send-capture.json. If it logs "template NOT picked", the account still has no online template — stop and create one first. Known unknown: the inner markup of the online container (its template picker and its send button's exact id) cannot be read today — AppFolio renders that container empty and shows the "No lease templates" warning in its place, so those selectors are only observable once a template exists. The harness therefore tries a selector ladder plus text matching and STOPS rather than clicking blindly; expect to adjust one or two selectors on the first real run, using the sibling naming convention visible today (#generate-pdf-template-lease-container / js-generate-pdf-template-lease-btn imply #generate-online-lease-container / js-generate-online-lease-btn). |
live-verify-cancel-module.ts | Verifies the shipped module end-to-end: raw-CDP Continue Move In to mint a live web_flow, then calls runCancelMoveInL4 with it. This is the proof behind stage 12 (returned status 200 on occ 1043). | npx tsx scripts/live-verify-cancel-module.ts — edit const OCCS. |
investigate-lease-templates.ts | Read-only survey of /lease_templates — what exists, whether templates are per-property or account-wide, and the create-form shape. This is how "only a Sample Lease Template exists, account-wide" was established. | npx tsx scripts/investigate-lease-templates.ts |
Gotchas for whoever picks these up: page.evaluate fails under tsx with "__name is not defined" (esbuild helper injection) — use locators and page.content() instead. Every authenticated AppFolio fetch must route through the browser page (setBrowserFetchPage); AppFolio's WAF binds session cookies to the Browserbase IP and a raw-Node fetch either 401s or gets soft-failed into a misleading 200-HTML re-render. All runs are scope-guarded to test property 45.