0098 — Executed-lease notice pipeline: decouple fetch from send, exactly-once outbox, PMS-agnostic acquisition

AMENDED 2026-08-13/14 (owner decisions, Fede) — the fetch never gives up, and waiting is SILENT. §3.1's 24h max-age wall, its 6h warn, and §7 Q1/Q2's AGED_OUT terminal are superseded: leases routinely take weeks or months to countersign — normal business, not an engineering condition — so the fetch workflow polls until the executed document exists (15m → hourly day 1 → 6-hourly week 1 → daily) with NO time-based alerts of any kind. The only page left is the wrong-document (content-gate reject) alarm. It never completes unfetched and never writes AGED_OUT; AGED_OUT survives as a manual-ops parking state only (inverse: revivePendingNotice). Driven by the 2026-08 incident: 11 real transfer executions countersigned late all aged out unsent — accounting silently received nothing (and the 11 age-out Sentry pages went unnoticed, proving time-based paging here is noise). See src/lib/temporal/workflows/executed-lease-fetch-workflow.ts (PRs #5741 + follow-up).

• Status: Proposed
• Date: 2026-07-16
• Deciders: Fede
• Supersedes / touches: ADR-0046 D3 (renewal execution notice), ADR-0081 G3 (occupancy transfer notices), the generalized notice in renewal-execution-notify.ts
• Context incidents: 2026-06-18 (wrong attachment — stranger's 2018 invoice), 2026-07-09 (false "signed" email, lease still out for signature), 2026-07-16 (attachment-less send on a transient runner timeout)

1. Context

When a lease is fully executed (signed + countersigned), the PMS notifies the office of nothing. PropFlow emails the property's accounting recipients a plain-English heads-up. This notice has broken in production three times in one month, each caught by the customer rather than by us.

Current architecture (verified in repo, 2026-07-16)

Three near-copy entry points all converge on the one shared send function notifyExecutionToAccounting (src/lib/domain/leasing/renewal-execution-notify.ts) — so the send template is already single-sourced. The breakage is not in the template; it is in the acquisition + orchestration wrapped around each caller:

All three, inline and under their own time budget, call fetchExecutedLeasePdf (renewal-orchestration/fetch-executed-lease-pdf.ts) → handleFetchLeasePdf handler → fetchLeasePdfL4 (Browserbase browser scrape of AppFolio, cold path ≈120s inline MFA) → content gate verifyExecutedLeaseContent (execution marker + tenant-name needle, fail-closed). Then they stamp, then send with the PDF attached.

The four root architectural flaws (confirmed with Fede)

1. PDF fetch is coupled to the send. The fetch runs under the sender's clock (60s activity / 300s cron). A cold fetch (~120s) cannot fit the 60s renewal activity at all; the cron only fits it by burning almost its whole budget on one candidate. A single transient runner blip (2026-07-16) drops the attachment.
2. Detection is single-channel. Only the Browserbase scrape of AppFolio confirms execution. No corroboration.
3. Three near-copy orchestration paths drift. The send template is shared, but the stamp semantics, retry policy, gate ordering, and PDF-fetch wiring are re-implemented per path and have drifted (three different bugs in three paths).
4. Link-only fallback degrades silently. notifyExecutionToAccounting sends deep-link-only when pdf is absent. #3954 turned that into a logCritical page, but the notice still goes out attachment-less.

2. Locked decisions (from Fede, 2026-07-16 — not relitigated here)

• L1. Never send a notice without the executed-lease PDF attached. Defer + retry + escalate (page) instead. No link-only sends, ever, for executed-lease notices.
• L2. PMS-agnostic core. The shared notice module must not import AppFolio anything. Executed-lease acquisition sits behind an interface. Yardi/RealPage adapters addable without touching the core.
• L3. S3 PDF cache, KMS-encrypted, private bucket, 48h lifecycle auto-delete. Attach bytes into the email — never a link to the object.
• L4. Acceptance gate: Willows end-to-end — a synthetic execution on the test property (appfolio-45) through the real prod chain, email opened with the PDF attached, as a repeatable canary.

3. Decision

Split the pipeline into detection → durable ingest (fetch+cache) → durable outbox (send), with a PMS-agnostic acquisition interface and an S3 byte cache between them.

             ┌─────────────────────── detection sources (dedup on noticeKey) ─────────────────────┐
             │  renewal D3 (Temporal)   prospect sweep   occupancy sweep   [staff-JSON:API]  [inbox-signal SPIKE] │
             └───────────────────────────────────────────┬───────────────────────────────────────┘
                                                          │ emitExecutedLeaseEvent(kind, anchorRef, source, payload)
                                                          ▼
                                        ┌──────────── Notice Outbox (DDB) ────────────┐
                                        │  PENDING_NOTICE row, conditional-put create   │
                                        │  status: PENDING_FETCH → FETCHED → SENT       │
                                        └───────────────────┬──────────────────────────┘
                                                            │ start/signal (dedup by workflowId)
                                                            ▼
                              ┌───────── executedLeaseFetchWorkflow (Temporal, propflow-renewal) ─────────┐
                              │  patient retry: fetch via ExecutedLeaseSource → content gate → S3 cache    │
                              │  generous per-attempt budget; backoff to hourly; max-age escalation (page) │
                              └───────────────────────────────┬───────────────────────────────────────────┘
                                                              │ on cached PDF → row FETCHED → sendPendingNotice
                                                              ▼
                        ┌──────────── sendPendingNotice (ONE module) ────────────┐
                        │  read outbox row → GET bytes from S3 → REFUSE if absent  │
                        │  → notifyExecutionToAccounting (attach bytes) → row SENT │
                        └──────────────────────────────────────────────────────────┘
                     (reconcile sweep retries FETCHED-but-not-SENT + re-drives stalled fetch workflows)

3.1 Ingest pipeline — decouple fetch from send

A dedicated executedLeaseFetchWorkflow on the existing propflow-renewal Temporal task queue (reuse the existing renewal worker — services/renewal-worker/index.ts — which already has AppFolio L4 access and the S3-capable task role; no new worker, no new cluster).

• Idempotency / dedup: workflowId = exec-lease-fetch-<pmsRef> (deterministic on the occupancy/lease foreign key). WorkflowIdReusePolicy: REJECT_DUPLICATE → a second detection (any source) for the same execution attaches to the running/completed workflow instead of spawning a duplicate fetch. This is the durable analogue of the current idempotencyKey scheme, promoted out from under the sender's clock.
• Patient retry: the fetchAndCacheExecutedLease activity gets startToCloseTimeout: 180s (room for a full cold MFA + multi-MB download) and Temporal activity retry (maximumAttempts high, exponential backoff). Between soft-failed attempts (lease not countersigned yet), the workflow sleeps and re-drives on a schedule (15 min → backing off to hourly) — a lease that is genuinely still out for signature is retried patiently for the whole max-age window without holding an activity open. Content-gate rejection is terminal within an attempt (a wrong document can't be re-fetched into a right one), but the workflow keeps polling because a later fetch may return the correct executed render (the 2026-06-18 wrong-namespace case resolves when the runner resolves the right doc).
• Success: cache the content-verified bytes in S3 (§3.3), transition the outbox row FETCHED, and call sendPendingNotice.
• Failure alerting (reuse #3954): a soft/transport failure per attempt → logError (Sentry, non-fatal). Max-age escalation: (superseded 2026-08-14 — see the amendment banner: no time-based warns/pages exist; only the wrong-document alarm remains) Max-age escalation (historical): if the workflow has not cached a PDF after N = 24h from first detection, logCritical (page) once, guarded by a conditional-put claim marker (§3.5) so a still-unfetched lease pages once/day, not once/tick (mirrors claimNtvEscalation in ntv-escalation-marker.ts). An earlier logWarn age-out at 6h (reuse warnIfNoticeAgingOut, notice-age-out-alert.ts) gives a soft heads-up before the fatal.
  • N = 24h rationale: a truly-executed lease is fetchable within minutes-to-hours once AppFolio sync + runner MFA succeed; 24h is comfortably past any observed transient runner/MFA/sync outage, yet still lands the page inside a single business day for a human to chase. 6h warn / 24h page — flagged as an open product question (§7 Q1) for Fede to confirm the paging sensitivity.

3.2 Notice outbox — exactly-once, defer-not-degrade

A durable PENDING_NOTICE DynamoDB item (single-table), created the moment any entry point detects a lease-executed event.

• Key: PK = PROP#<propertyId>, SK = PENDING_NOTICE#<noticeKey>, where noticeKey = <kind>#<anchorRef> (anchorRef = occupancy pmsOccupancyRef for move-in/transfer, saga id for renewal). Deterministic → the dedup anchor.
• Fields (ids only — no tenant names, no bodies; PII resolved at send time from the Person spine): kind (renewal|new_move_in|transfer), personId, occupancyRef, prospectId?, sagaId?, pmsPropertyId, detectionSources: string[] (append-on-dedup), pdfS3Key?, pdfCachedAt?, status (PENDING_FETCH|FETCHED|SENDING|SENT|AGED_OUT), claimedAt?, sentAt?, attempts, lastError?, payload? (renewal newRent/termMonths), createdAt, ttl.
• Create is idempotent: conditional-put attribute_not_exists(SK); a duplicate detection instead appends its source to detectionSources (dedup + provenance) and is otherwise a no-op.
• Send is claim-guarded (no double email): sendPendingNotice first claims the row FETCHED → SENDING (conditional status = FETCHED, stamping claimedAt) before it touches the mailer — a concurrent drainer's claim throws and it returns without sending. The claimer then mails and completes SENDING → SENT (conditional status = SENDING), or rolls back SENDING → FETCHED on a soft failure so the sweep retries. The claim, not the mailer, is the double-send guard: the mailer (notifyExecutionToAccounting) has no send-time idempotency, so a bare FETCHED → SENT guard would only stop a double transition after both drainers had already mailed. A claimer that crashes mid-send leaves a stale SENDING row (claimedAt older than the stale window) that the reconcile sweep re-queues — the standard outbox at-least-once tail.
• Send gate = the invariant: sendPendingNotice reads pdfS3Key, GETs the bytes from S3, and refuses to send if the key is absent or the object is gone (returns a defer, does not degrade). This is where L1 is enforced structurally — the send module physically cannot send without bytes in hand.
• Reconcile sweep (new lightweight Vercel cron, replacing the two heavy fetch-inline sweeps over time): retries FETCHED-but-not-SENT rows (transient email transport failures), re-queues stale SENDING rows whose claimer crashed mid-send (claimedAt older than the stale window → roll back SENDING → FETCHED), and re-drives outbox rows whose fetch workflow is missing/stalled. No PDF fetch happens in this sweep — it is fast (no 150s soft-budget dance needed).

Migration — do NOT strand in-flight state (2026-06-24 lesson). The outbox is purely additive:

• The legacy stamps (ProspectInquiry.executionNoticeSentAt, TenantOccupancy.transferNoticeSentAt, renewal alreadyDispatched) keep being read as "already sent" throughout migration, so nothing double-sends.
• During each path's cutover PR, the send module dual-writes both the outbox status=SENT and the legacy stamp, so a rollback still sees the stamp and the old sweep still no-ops.
• No backfill of in-flight sagas/occupancies — we never touch a live record's saga state (the 2026-06-24 dropped-decline class of bug). Old in-flight rows finish on the legacy path; new detections take the outbox path.
• Legacy-stamp reads are deleted only in a later cleanup ADR once all pre-cutover rows have drained (>30d), mirroring the ADR-0092 CONSENT# dual-read teardown.

3.3 S3 executed-lease cache

Reuse the propflow-photos bucket with a new executed-leases/ prefix — not a new bucket. Justification vs the evidence-bucket precedent (which did need a separate bucket): evidence needed Object-Lock, which can only be set at bucket creation and is incompatible with our requirement here (we want auto-delete at 48h, the opposite of immutable retention). Nothing about this cache needs a dedicated bucket; a prefix + a prefix-scoped lifecycle rule gives lifecycle isolation without new IAM surface. The renewal worker task role already holds s3:PutObject/s3:GetObject on propflow-photos/renewal-letters/* (reference: renewal_worker_task_role_s3); extend the resource allowlist to propflow-photos/executed-leases/*.

• Key shape: executed-leases/<propertyKeyPrefix>/<noticeKey>.pdf (reuse propertyKeyPrefix() from s3/client.ts).
• Module: src/lib/integrations/s3/executed-lease-cache.ts, modeled on evidence-bucket.ts — putExecutedLeasePdf, getExecutedLeasePdf, buildExecutedLeaseKey, isExecutedLeaseCacheConfigured (env-guarded; degrades in preview/disarmed).
• 48h lifecycle: an S3 lifecycle rule scoped by Filter { Prefix: "executed-leases/" }, Expiration { Days: 2 } — does not touch photos/ or renewal-letters/. (S3 lifecycle min granularity is 1 day; "48h" = Days: 2.)
• KMS: must verify whether propflow-photos default encryption is SSE-KMS or SSE-S3. If SSE-S3, either enable bucket-default SSE-KMS or set per-object ServerSideEncryption: 'aws:kms' + SSEKMSKeyId on these writes. Flagged infra verification (§7 Q6) — L3 requires KMS.
• Access logging: enable S3 server access logging (or CloudTrail data events) on the bucket for these objects.
• Attach = bytes, never a link (L3): sendPendingNotice GETs the object into a Buffer and passes it as the email attachment. The object URL never leaves the backend.

3.4 PMS-agnostic acquisition interface (L2)

Define ExecutedLeaseSource in the domain (not in integrations/):

// src/lib/domain/leasing/executed-lease-source/types.ts
export interface ExecutedLeaseFetchInput {
  pmsRef: string;            // opaque FK (AppFolio: occupancyId; Yardi: propertyId+leaseId)
  pmsPropertyId: string;
  externalLeaseDocId?: string;
  tenantIdentity: { firstName: string; lastName: string }; // content-gate needle
  idempotencyKey?: string;
  timeoutMs?: number;
}
export interface ExecutedLeaseDocument { pdf: Buffer; extractedText?: string; filename?: string; }

export interface ExecutedLeaseSource {
  readonly pmsType: PMSType;
  /** Return the content-VERIFIED executed lease, or undefined if not executed / unverifiable.
   *  The adapter runs the PMS-agnostic `verifyExecutedLeaseContent` gate before returning. */
  fetchExecutedLease(input: ExecutedLeaseFetchInput): Promise<ExecutedLeaseDocument | undefined>;
  /** Optional corroboration channel (phase 7): a cheaper/second signal that the lease executed. */
  corroborateExecution?(input: { pmsRef: string; pmsPropertyId: string }): Promise<{ executed: boolean; source: string }>;
}
export function getExecutedLeaseSource(pmsType: PMSType): ExecutedLeaseSource; // registry, dispatch by pmsType

• AppFolio adapter wraps the existing fetchExecutedLeasePdf (L4 + verifyExecutedLeaseContent) — no logic rewrite, just re-homed behind the interface. It is the only place AppFolio ids/L4 are referenced. This also aligns with the existing optional PMSWriter.fetchExecutedLeasePdf?(FetchExecutedLeasePdfParams) in src/lib/domain/pms/writer.ts — the new interface is the acquisition peer of that writer method; the AppFolio adapter can delegate to it.
• verifyExecutedLeaseContent stays a pure, PMS-agnostic function in core (already is) — the content gate is not PMS-specific.
• Yardi SIPP adapter — signatures only, no impl (illustrates the seam):

// src/lib/integrations/yardi/executed-lease-source.ts  (SKELETON — not implemented)
export class YardiExecutedLeaseSource implements ExecutedLeaseSource {
  readonly pmsType = 'yardi' as const;
  async fetchExecutedLease(input: ExecutedLeaseFetchInput): Promise<ExecutedLeaseDocument | undefined> {
    // SIPP ItfCommonData GetLeaseDocuments-family call → select the executed/countersigned doc
    // → decode → verifyExecutedLeaseContent({ text, tenantIdentity }) → return or undefined.
    throw new Error('not implemented');
  }
}

3.5 Telemetry invariants (carry forward #3954 / #3953)

• Attachment-less send is now structurally impossible for executed-lease notices: sendPendingNotice refuses without bytes. The #3954 logCritical "sent WITHOUT lease PDF" degradation path in notifyExecutionToAccounting is converted from a post-send page to a pre-send refuse-and-defer for this category (Phase 6). The critical signal moves to fetch-exhaustion (max-age N) instead of attachment-less-send.
• Pending-notice age-out escalation / fetch-exhaustion page — removed 2026-08-14 (amendment banner): a still-waiting lease emits no alert at any age; notice-age-out-alert.ts is deleted.
• The invariant test (one test, all entry points): a single test enumerates every emitter and asserts that no path can reach notifyExecutionToAccounting for an executed-lease notice without a content-verified PDF. Concretely: (a) sendPendingNotice given an outbox row with no pdfS3Key never invokes the email transport; (b) an architectural assertion that the three emitters call emitExecutedLeaseEvent and never notifyExecutionToAccounting directly.
• No bodies in logs (#3953): the outbox row and all new logs carry ids only; EMAIL_LOG body exclusion unchanged.

3.6 Multi-source detection (design only; built in Phase 7)

emitExecutedLeaseEvent(kind, anchorRef, source, payload) accepts a source enum: renewal-workflow | prospect-sweep | occupancy-sweep | staff-jsonapi | inbox-signal. Dedup on noticeKey; multiple sources for one execution append to detectionSources and produce one notice.

• Staff JSON:API corroboration (cookie-auth internal AppFolio surface — reference appfolio_staff_jsonapi): a cheaper second signal (ExecutedLeaseSource.corroborateExecution) that the lease executed, reducing the 07-09 false-signed class. Phased later.
• Inbox-signal — SPIKE only. It is UNVERIFIED whether AppFolio emails the office on execution. Marked as a research spike; do not build on the assumption. If confirmed, the inbound-email path (extractOperationalSignal in webhook-processors.ts) becomes another emitter — subject to ADR-0097 (email signals never create records; they'd only corroborate an existing outbox row).

4. Consequences

Positive: fetch runs on its own patient clock (root flaw 1 gone); one orchestration module owns send (flaw 3 gone); link-only is structurally impossible (flaw 4 gone); detection is pluggable + multi-source-ready (flaw 2 addressed); PMS-agnostic core unblocks Yardi/RealPage; 48h KMS cache satisfies retention/PII.

Negative / cost: a new DDB item type + a new Temporal workflow + prod worker deploy (daylight-gated); a migration window with dual-write; more moving parts than three inline sweeps. Mitigated by additive migration (no behavioral gap) and by reusing the existing worker/queue/bucket/task-role (no new infra footprint beyond one S3 prefix + lifecycle rule + IAM statement).

Neutral: the two heavy cron sweeps shrink to a light reconcile sweep once all paths cut over.

5. Alternatives considered (rejected)

• Just widen the sender's time budget. Rejected — a 120s cold fetch can never fit the 60s renewal activity, and widening the cron budget still couples fetch to send and keeps three drifting paths.
• New dedicated S3 bucket (like evidence). Rejected — evidence needed Object-Lock (create-time only); we need auto-delete. A prefix + prefix-scoped lifecycle rule is lighter and needs no new bucket/IAM.
• SQS instead of a Temporal workflow for ingest. Rejected — the patient, schedule-with-backoff, max-age-escalation retry loop is exactly what Temporal expresses natively, and the renewal worker + queue already exist. SQS would re-implement the durable timer.
• Keep link-only as last-resort after N hours. Rejected by L1 — never send without the PDF; page + hold instead.

6. Phased implementation plan (per-PR scopes)

Ordering guarantees: one concern per PR; the two current sweeps + renewal D3 keep sending throughout; no window where notices stop. Each PR ships regression tests + a Willows E2E step.

Tonight-sized: 1, 2, 3, 6. Needs daylight (prod worker deploy / AWS infra / live prod cutovers with Willows E2E each): 0, 4, 5a/5b/5c, 7, 8.

No-gap guarantee: Phases 0–4 are all dark/additive — the three current paths send exactly as today. Each Phase-5 PR flips exactly one path and dual-writes its legacy stamp; the other two keep their current behavior until their own PR. At no point do notices stop.

7. Open product questions (NOT locked above — need Fede)

1. Max-age escalation N. Proposed 6h warn / 24h page. Confirm the paging threshold and business-hours sensitivity (page at any hour, or defer to next business morning?).
2. What happens at N-hours-unfetched. Proposed: never auto-send link-only; page + hold the outbox row (AGED_OUT) until a human fetches manually or dismisses. RESOLVED 2026-08-13/14 (Fede): keep polling forever, silently. There is no N — a still-uncountersigned lease at any age is normal business and emits no alert; the workflow only completes when the executed document is fetched. Link-only auto-send remains forbidden (unchanged).
3. Corroboration-before-send. Is a content-gated PDF alone sufficient proof to send (my lean — the verified executed PDF is the proof), or should a second detection source (staff-JSON:API) be required before send to kill the 07-09 false-signed class outright?
4. 48h TTL vs human-in-the-loop window. If a PDF is cached at hour 40 and a human acts at hour 47, the object may already be near expiry. Is 48h enough, or should the clock be "48h after cache" (per-object) rather than a blanket prefix rule? (Per-object expiry needs an object tag + tag-scoped rule.)
5. Renewal payload parity. Renewal D3 sends newRent/termMonths (executed term); move-in/transfer omit them. Confirm the outbox payload should carry these for renewals only (not a blocker — defaulting to omit is safe).
6. KMS on propflow-photos. Infra verification, not a product call, but must be resolved before Phase 0 ships: is the bucket SSE-KMS today, or does this need per-object KMS / a bucket-default change? (L3 requires KMS.)
7. Legacy-stamp teardown timing. Coexist through migration, delete legacy reads in a follow-up ADR after >30d drain (ADR-0092 pattern). Confirm the drain window.