Inbound email: what dropped Denise, and the deterministic-first redesign

TL;DR

A real Zillow lead (Denise Holguin, Camellia) was silently dropped by the inbound-email pipeline on 2026-07-24. She was recovered same-day through the real prod path — Clara replied at 4:44 PM MT. The investigation found the drop is systemic, not an edge case: 76% of 8,182 prod ingestion rows end in silent skips, and policy_skip is an unalerted, unrecoverable sink fed by at least five independent mechanisms.

A redesigned pipeline — deterministic-first, LLM only for genuinely ambiguous human free-text — was prototyped and benchmarked against the current pipeline on 50 real prod emails, 5 trials each:

Under an injected total LLM outage the prototype still replies to both guest-card leads; the current pipeline answers neither. One earlier causal claim did not survive the experiment — see the correction below.

Head-to-head results 50 emails × 5 trials × 3 arms

Named failures (no bare “passed”)

• Prototype, 2/50 wrong: “Re:Parking” (read as current resident; corpus itself flags it ambiguous) and “Please print” — misdirected HR mail that should have landed in needs_review but got a high-confidence operational_extract. That one is the real miss.
• Current, 5/50 wrong (both menus): four are real operational mail discarded as spam by AI triage — a staff utility bill, both sides of a syndication support case, a calendar invite. A silent loss through a different branch than the one that dropped Denise.

Denise & Jay guest cards, per arm

All three arms landed reply / tour_request 5-of-5 in the harness. The difference is mechanism and cost: the current pipeline spends 10 LLM calls across the pair and the answer is a sampled token — prod sampled differently on 2026-07-24 and never replied to Denise. The prototype resolves both at zero LLM calls via the guest-card body parser. With a dead LLM endpoint injected, the prototype still replies to both; the current pipeline files both as operational and answers neither.

Correction: the contamination link is not supported

The systemic finding stands regardless of which mechanism stamped the row: an open classification vocabulary feeding an exact-string response allowlist means any unrecognized label — from contamination, LLM flake, keyword fallback, or a novel-but-valid category — becomes a silent drop.

Flaw inventory from 3 adversarial reviews

• stripQuotedReply returns an empty string for any reply starting “On <date>…” and for forwarded messages → classified unknown → dropped. Verified by executing the real regex.
• No safety-net poller exists. The functions are dead code with zero callers; comments claiming “the poller is the safety net” are stale. The webhook is the only path and Graph already got its 202 — every drop is permanent.
• A Graph 429/5xx on message fetch returns null — no throw, no requeue. The email is gone with only a log line.
• classifyEmail has no resilience wrapper while the triage call beside it in the same Promise.all does — one LLM hiccup routes a real prospect to a non-allowed classification.
• Concurrent webhook notifications clobber webhookSeenIds via unguarded read-modify-write, and can persist a superseded OAuth refresh token for the whole mailbox.
• Observability is thin: classificationSource present on 1.3% of rows, agentTraceId on 9.7%.

The redesign: deterministic-first, LLM-last

Design directive (Fede, locked): maximize determinism — the LLM is a fallback for genuinely ambiguous human free-text, never the default path. Low confidence goes to a human, never to a guessed category.

flowchart LR
  A[Inbound email] --> B{Sender-class router
deterministic}
  B -->|guest card / aggregator /
system mail / bounce / internal| C[Closed category
0 LLM calls]
  B -->|structured body| D[Parsers
guest-card, portal relay]
  D --> C
  B -->|unknown human sender,
free text| E[One structured LLM call
category + confidence]
  E -->|high confidence| C
  E -->|low confidence / failure /
empty-after-strip| F[needs_review
never drops]
  C --> G[Action map over closed enum
unmapped = compile error]
  F --> G
    

• Closed category enum — a typed union; the DB-derived vocabulary (getExistingCategories) is deleted. No data can widen the label space.
• One structured decision call replaces the triage+classifier pair, killing their contradiction modes; forced tool output, returns {category, confidence, reasoning}.
• needs_review is a first-class terminal state that never drops — low confidence, LLM failure, empty-after-strip, and unknown sender class all land there.
• Response policy inverted to an exhaustive action-map over the enum. An unmapped category is a TypeScript compile error (verified: deleting a member produces TS2741), not a silent runtime drop.
• Decision-level persistence — every email gets a durable decision record with its route, category, confidence, and terminal state.
• Observability as drift/rate alerts (needs_review rate, per-category volume), not per-email pages.

Honest caveats

• In-sample bias: the deterministic rules were written with the 50-email corpus visible, so 66% zero-LLM and 96% correct are in-sample numbers. A held-out corpus should be scored before shipping claims.
• The confidence net barely fired in the main run (high confidence 249/250). Its plumbing is proven only by fault injection: with a dead LLM endpoint, 17 emails landed in needs_review, zero dropped, and both guest cards still got replies.
• Harness ≠ prod: the current-pipeline arm is a faithful adapter over the real modules, but it is not the deployed webhook path end-to-end; prod-only behaviors (Graph fetch failures, concurrency) are covered by the flaw review, not the A/B numbers.
• The harness never reproduced the Denise drop, which is itself informative: the failure lives in low-probability sampling or in a mechanism outside the replayed modules (see the correction above).

Decisions needed & next steps

• decide Green-light the security P0 PR (sender-trust anchoring + DMARC gate) — held for explicit approval.
• decide Approve productionizing the deterministic-first pipeline (closed enum, action-map, needs_review queue + its UI surface) as a staged rollout behind the existing property scoping.
• verify Resolve the appfolio_ntv_processed provenance question — what actually stamps that string on prod rows.
• verify Score the prototype on a held-out corpus the rules never saw.
• quick win Fix stripQuotedReply emptying “On <date>…” replies, and add requeue-on-fetch-failure — shippable independently of the redesign.

Artifacts: ~/.claude/propflowai-ingest-harness/harness/REPORT.md · immutable results in harness/results/final/ · prototype in harness/prototype/. Branch is local-only by design; nothing has been pushed or merged.