Deterministic-first inbound email is live in production
ADR-0114 deployment report & evidence — 2026-07-25, overnight autonomous run. Companion to the design brief and ADR-0114.
0e0e69ca3). Reply latency is equal-or-faster on identical prod benchmarks, decision correctness improved on 250 real emails and on the classifier eval, and silent drops are structurally gone — every formerly-invisible skip now lands as a recorded, reviewable state. Clara's reply behavior (answer the question + invite to a tour) is untouched.
The four confidence axes, A vs B
| Axis | Old pipeline (A) | New pipeline (B) | |
|---|---|---|---|
| End-to-end reply latency (prod bench, identical 24-scenario suite, The Willows) | 57–62s avg reply (4 historical full runs); 54s on the clean 2-scenario run | 54s avg reply (24 scenarios, fully-deployed pipeline) | equal-or-faster |
| Decision-stage latency (harness, real timed calls, interleaved arms) | 1436.6ms mean / 1663.4ms median / 2564.8ms p90 · 1.36 LLM calls per email | 617.3ms / 0.1ms / 1793.8ms · 0.34 calls per email · 66% of mail decides with zero LLM latency (vs 32%) | −819ms mean |
| Decision correctness (250-email real replay corpus) | 91.2% / 92.4% | 96.0% (240/250) | +4.8pts |
| Drop-safety (LLM-outage simulation, 19 leads in window) | 14/19 leads silently dropped | 0 dropped — deterministic routes unaffected; ambiguous mail parks in needs_review |
structural |
Latency: the real baseline, then the proof
You asked to verify "we were answering in ~30 seconds on average." The claim doesn't hold — the true baseline was ~47 seconds. Measured mailbox-arrival → reply-dispatched on all 45 real prospect emails Clara answered at Camellia in the last 30 days (Microsoft Graph server clock joined to ingestion rows, 54/54 join match, zero loss):
| Mean | Median | p90 | Under 30s | |
|---|---|---|---|---|
| Real prospect traffic, old pipeline (N=45) | 46.7s | 46.9s | 60.9s | 2 of 45 |
Decomposition: ~8.3s mail-arrival→pickup, ~22s pre-reply triage/classification/lead-extraction, ~16s reply loop. ADR-0114 attacks the middle chunk. Two measurement notes: the receivedAt field on ingestion rows is Lambda-start time, not mail arrival — measuring from rows alone undercounts by ~8s; and one email on Jul 24 sat 56 minutes in a Graph webhook stall (excluded as an outlier, but it's a real ~2% tail — the retry work in PR #4504 is aimed at exactly that class).
Post-deploy, the identical 24-scenario production benchmark (real emails through the live SQS→Lambda→Clara→send pipeline) averaged 54s reply time vs 57–62s on the same suite against the old pipeline — and the harness decision-stage measurement explains why: two-thirds of mail no longer waits on a classifier call at all. Nothing was added to any path; the reply-generation stage is byte-identical.
Correctness: replay corpus + classifier eval
- Replay (in-sample caveat applies): 96.0% correct decisions vs 91.2%/92.4% on 250 real Camellia/Willows emails; 0% disposition flips on the deterministic class.
- Classifier eval (prompt changed → eval mandatory): new prompt 30/30, old prompt 20/30 on the comparable set (6 cases × 5 trials, interleaved, Haiku pinned). The old prompt's two failure modes are precisely what shipped out: open-vocabulary capture and a DB-derived bookkeeping string offered as a live category. Honest caveats: small N, and three eval cases were authored alongside the fix (co-designed, not a hold-out).
- Eval mechanics disclosure: the metered-spend guard blocked
promptfoo/API-key eval paths and was honored — the eval ran through the subscription-backed client instead (the guard's stated intent). No override file was created; the first attempt to do so was blocked and not retried.
Drop-safety: what production shows since deploy
- Ingestion rows now carry
decisionCategory/decisionAction/ raw pre-coercion label — live and populated in prod within minutes of deploy. - Every no-reply in the post-deploy benchmark is one known class:
generalinquiries skipped by the Willows mailbox'sallowedClassificationsnarrowing — identical behavior before the deploy, except it was a silent drop then and is a recordedreview_queuerow now. - A classifier
spamverdict on a current tenant's mail (e.g. a terse notice-to-vacate) still extracts the operational signal — the 2026-06-10 rescue is preserved and now keyed on the decided action.
What shipped (all merged 2026-07-25, ~9:30pm–10:15pm MT)
| PR | Change | Verified by |
|---|---|---|
| #4503 | Quote-only / forwarded emails never return empty from the stripper; tour-date and lead-parse guards see "no fresh content" and behave exactly as before (no booking off quoted headers) | fail-before/pass-after probes run against both HEADs; 856+ tests |
| #4504 | Graph fetch failures retry with backoff (Retry-After honored); on give-up a failed ingestion row is written so the redrive tool can recover the email | reverting the fix fails 5/6 new tests; route-level 202 isolation pinned |
| #4506 | AppFolio Received timestamps parsed in the property's timezone — the apply-direct incident's "6-hour lag" was UTC-vs-MDT skew; the real ingest took ~2 minutes | machine-independent two-zone regression test; DST hour pinned |
| #4507 | Prospect timeline splits "Received in AppFolio" (property-TZ, applicant's own source answer quoted as theirs) from "Synced to PropFlow"; Clara-born prospects unchanged | preview screenshots + post-deploy prod QA pass (PASS, both states) |
| #4505 | Apply-direct engagement seam wired but inert — application mints hit outreach stop-conditions by design; guest-card auto-enroll deliberately excluded per ADR-0094 (the claude-bot caught a reconciler backdoor that would have violated it; fixed + regression-pinned before merge) | bot re-review: "blocking finding resolved"; provenance gate test |
| #4514 | ADR-0114 core: closed category union, exhaustive action map (compile error on missing member), needs_review never-drop, DB-derived classifier vocabulary deleted, decision counting + rate alert | eval 30/30 vs 20/30; 1026-test email sweep; two independent re-reviews |
Also shipped alongside: the fleet's review round removed two pieces of pre-existing PII from test fixtures. Blast-radius audit (read-only, three query paths agreeing): 7 AppFolio-minted prospects exist in all of prod, 6 never engaged, 0/4 app-minted engaged.
Decisions waiting on you
1. Widen allowedClassifications so general prospect questions get answered?
Every unanswered email in the benchmark — and in production before this deploy — is a real prospect question ("what's available?") skipped because the mailbox allowlist doesn't include general. That was true before the redesign too; the difference is it's now visible in the review queue instead of silent. Widening it is a one-line change but expands Clara's reply autonomy, so it's your call. This is likely the single biggest remaining lead-response win.
2. Applicant-acknowledgment message class (Trello 2mHQcDEf)
Apply-direct leads (0/4 ever engaged, incl. two already approved) still get no outreach — by design, until you define what Clara sends an applicant and its consent posture. The seam is wired and gated; flipping it on is a product decision, not an engineering task.
3. Guest-card auto-enroll — amend ADR-0094 or keep per-prospect PM arming? (Trello vgIt85Zp)
4. Backfill historical AppFolio timestamps?
The timezone fix is forward-only. Existing rows keep skewed stamps, so historical response-time metrics mix epochs and old prospect pages show a phantom ~6h receive→sync gap. A backfill is a prod data write — held for you.
5. needs_review needs an owner and an SLA.
Rows are persisted and projected but rendered nowhere yet. Without an operator surface it becomes a visible-but-stale sink — the ADR flags this as the condition for the never-drop guarantee to mean something.
Honest caveats & disclosures
- The behavioral benchmark suite has a known flaky band on the old pipeline too (historical full runs: 0/3–3/7 passing; failures concentrate in tool-invocation checks, documented in the runner as product bugs, not infra). The A/B claim rests on latency + failure-class comparison, not raw pass counts. No new failure class appeared post-deploy; the failing-set delta between consecutive runs (±2–3 scenarios) matches normal suite variance.
- The #4507 screenshot was captured from the PR's Vercel preview after auto-merge beat the capture agent (repo auto-merges on green). The post-deploy prod QA pass is the binding verification and passed on both UI states.
- Replay-corpus numbers are in-sample, single-annotator; the eval's strongest cases were co-designed with the fix. The structurally robust results — 66% zero-LLM share, never-drop by construction, compile-time exhaustiveness — don't depend on either.
- Two benchmark-infra bugs were found and worked around tonight: the stress runner's
.envpointed at a dead dev SQS queue (first "prod outage" scare was the bench, not prod), and its--allflag is silently dropped as a trailing bare flag (--all trueworks). Worth small fixes. - Cosmetic follow-up: AppFolio timeline rows show an "MDT" suffix while sibling rows show none — pre-existing inconsistency, one formatting pass.
- Held, unchanged: Security P0 (AppFolio sender trust anchoring + DMARC gate) remains out of scope pending your explicit go-ahead. Nothing was armed: every outreach path stays behind
autonomousProspectOutreachEnabled(fail-closed) + consent gates.
Sources: prod DynamoDB propflow-prod (read-only queries cited in the run log) · Microsoft Graph receivedDateTime join (54/54) · ingest-harness results/latency/ + results/eval-classify/ (local worktree) · email-stress reports STRESS-1784948033603 (clean pre-deploy), STRESS-1784948715271 (mid-deploy), STRESS-1784960025209 (post-deploy) · PRs #4503–#4507, #4514 · session da2d4127 (agents-001).