PropFlow.ai PropFlow.ai Docs
Acceptedmarkdown source

ADR-0114: Deterministic-first inbound-email decision pipeline — closed categories, exhaustive action map, never-drop review state

Context

A real Zillow → AppFolio guest-card lead (Camellia, 2026-07-24) was silently dropped: no reply, no alert, no recovery path. The audit found the drop is structural, not an edge case — 76% of 8,182 prod ingestion rows end in silent skips, and of the 14 classification strings the classifier has actually emitted, 11 fall through shouldRespond's exact-string allowlist into policy_skip.

The structural defects (full inventory and evidence in the audit doc):

  1. Open classification vocabulary. The classifier is fed every classification string ever written to the DB (getExistingCategories()), so the label space grows with data and nobody decides what new labels should do.
  2. Drop-by-default response policy. shouldRespond is an allowlist over that open vocabulary; anything unrecognized — LLM flake, keyword fallback, novel-but-valid label — silently drops. The policy cannot distinguish "category we chose not to answer" from "string nobody has seen".
  3. Two LLM calls that can contradict (triage + classify); in the harness, triage deleted real operational mail as spam (4 of the current pipeline's 5 corpus failures).
  4. No "not sure" destination — ambiguity must become a guess, and a bad guess becomes a drop or a deletion.
  5. LLM in the hot path for machine mail — guest cards and system notifications ride two sampled LLM calls; the incident was one bad sample on an email that is a template.
  6. No recovery — webhook-only ingestion (post-ADR-0011), Graph fetch failures return null with no requeue, and the empty-body strip bug (stripQuotedReply) feeds the drop sink.

Replay-harness evidence (50 real prod emails × 5 trials × 3 arms; details, named failures, and caveats in the audit doc): a deterministic-first prototype scored 96.0% correct disposition vs 91.2–92.4% for the current pipeline, resolved 66% of emails with zero LLM calls vs 32% (0.34 vs 1.36 calls/email), had zero disposition flips across repeat trials, and under a forced total LLM outage escalated instead of dropping (current pipeline silently filed 14 of 19 real leads; prototype dropped 0). Numbers are in-sample — the rules were written with the corpus visible — and ground truth is single-annotator; treat them as a signal, not a measurement.

Decision

Rebuild the inbound-email decision layer deterministic-first, LLM-last:

  1. Closed category set. Email categories become a typed union in code. getExistingCategories() and the DB-derived vocabulary are deleted. No runtime data can widen the label space.

  2. Deterministic sender-class router runs first. Every known machine sender resolves with zero LLM calls: AppFolio guest cards (template-parsed — routing depends on structure, not on the prospect writing free text), lead aggregators, AppFolio system mail, internal domains, transactional platforms, calendar invites, bounces/auto-replies. Cheap deterministic content rules (DSN/auto-submitted headers, empty-after-strip) also resolve before any model call.

  3. One structured LLM decision call for the residual (unknown human sender, free text), returning {category, confidence, reasoning} via forced tool schema. The triage/classify pair is deleted; spam is an enum member with a decided action, not a separate veto.

  4. needs_review is a first-class terminal state that never drops. Low confidence, LLM failure, empty-after-strip, and unknown sender class land there, alerted and human-visible. Because LLM self-reported confidence proved uncalibrated (high-confidence 249/250 in the harness), the trigger must not rely on stated confidence alone — route unknown-sender mail with no category-specific evidence to review regardless.

  5. Response policy inverted to an exhaustive action map over the closed enum with no default arm. A category without a decided action is a TypeScript compile error (TS2741), not a silent runtime drop. The policy_skip failure mode becomes inexpressible.

  6. Decision-level persistence + drift alerts. Every email gets a durable decision record (route, category, confidence, terminal state); observability is rate/drift alerts on needs_review and per-category volume, not per-email pages.

  7. Categories assert structure, not intent — and reply behavior is unchanged. The deterministic router labels what the email is (e.g. a prospect inquiry from an aggregator guest card), never what the prospect wants; sub-intent (pet policy, pricing, tour ask) is read from the body by Clara's LEASING conversation layer. Clara's existing prospect reply behavior — answer the asked question AND invite to a tour — is deliberately preserved as-is: the funnel goal is tours, then applications from those tours. This redesign changes which emails reach Clara, not what she says.

Rollout: harvest-first. The structural fixes (closed enum, action map, needs_review queue + surface) land in the current pipeline before any larger rewrite; independent quick wins (stripQuotedReply empty-string bug, requeue-on-Graph-fetch-failure, silent-sink rate alerts) ship immediately as separate PRs. Before autonomy claims, the router/decision layer must be re-scored on a held-out corpus the rules were not written against.

Out of scope here, decided separately: the held security P0 (AppFolio sender-trust anchoring + DMARC gate) — tracked in the audit doc §2.1 and held for Fede's explicit go-ahead because it changes sender trust on the live Camellia mailbox.

Consequences

Alternatives considered