PropFlow.ai PropFlow.ai Docs
**Reverted** 2026-04-27. The scatter-gather architecturemarkdown source

ADR-0010: Extract the attachment parser into its own Lambda

Reversal — 2026-04-27

Decision: Revert Phase 8. Bundle the rent-roll / PDF / XLSX / PAR parsers back into Clara's Lambda. Parse inline. Delete the attachment-parser Lambda and the orchestrator scaffolding.

Why we reverted, in order of severity:

  1. Five distinct production incidents in one day, all rooted in the scatter-gather coordination layer:

    • Phase 8.4 cutover guard tripped on a Yale email because env vars were stripped without scatter being engaged (the original "Attachment parsers unavailable" failure that motivated this whole thread).
    • The deploy script wiped env vars on every update (preserved-on-update fix landed in PR #431 mid-incident).
    • Lambda version snapshot drift: :live alias pointed at v180 which had no orchestrator env vars, while $LATEST had them. The prod-promote workflow froze the wrong config into the version.
    • Phase 8.3 (real-traffic handler) was opened as PR #431 on 2026-04-26 but never merged. Phase 8.2 (skeleton that throws on real traffic) ran in prod for ~24 h with scatter routing live, sending every attachment to a Lambda that immediately failed.
    • The DDB-counter / S3-PutObject race (Promise.all of decrement + write) — caught in /simplify on 2026-04-27, but the fact that this was a real heisenbug shows the coordination layer is bug-prone, not bug-free.

  2. Zero functional gain. The original motivation was Lambda cold-start latency and bundle size for the conversational-email common case (95%+ of traffic). At current volume (~10 emails/day, single-tenant property roll-out), Clara's cold start is sub-second already; the ~3 MB bundle delta from inlining the parsers is invisible against per-invoke variance. We paid real failure-mode cost for a theoretical performance win we cannot measure.

  3. Operator burden. Three env vars × two Lambdas × two version-pinned aliases = many places where a single config drift breaks the data pipeline silently. Each incident in (1) was a different shape of the same problem: state distributed across systems that didn't need to be.

  4. The simpler shape works. Pre-Phase-8 Clara had processInboundEmail call processAllReportAttachments inline — one function, one Lambda, one failure mode (parser threw). It fit on one screen. It is the shape we are returning to.

What stays from Phase 8 work:

When this could come back. If we hit ≥100 emails/minute sustained and Clara's cold start regresses past ~5 s and the parsers specifically (not the agent loop) are the bottleneck, an out-of-band parser is the right reach. We will know the numbers when we see them; until then, inlining is the default.

Lessons recorded for future architecture decisions:

The original decision text follows for history.

Context

When a prospect or PM sends an email with a data attachment, Clara's Lambda (propflow-inbound-processor) currently parses the attachment inline on the same compute that is running the leasing/maintenance agent loop:

  1. SES → SQS → Lambda handler receives the SES message.
  2. The handler fetches the raw email from S3.
  3. processInboundEmail runs simpleParser, uploads attachments to S3, then on the SAME invocation runs every applicable parser:
    • parsePARReport (Claude Vision on the PDF — ~50 seconds)
    • parseWeeklyReport (XLSX → structured JSON)
    • detectAndParse (rent-roll detect + LLM fallback)
    • parsePdfWithVision (rent-roll PDF fallback — another Vision call)
  4. Only after all parsing finishes does the agent loop run against the extracted content — or against the plain body text if it's a conversational email.

This design has a few problems:

1. Latency coupling

A conversational email (common case, 95%+ of traffic) pays nothing for attachment parsing — but the Lambda still bundles all the parser code and its dependencies. Cold-start penalty for everyone.

2. Bundle bloat

The inline parsers pull in pdf-parse, xlsx, papaparse, plus the full @/lib/rent-roll/** and @/lib/property-reports/** trees. These add multiple megabytes to Clara's Lambda bundle, which already bumped up against the 50 MB zipped limit during the Phase 4 extraction.

3. Timeout risk

Vision-based parsing takes 30-60 seconds per PDF. An email with two PDFs can run for two minutes in Clara's Lambda — but the handler has to also do agent work, send a reply, and flush traces before the 15-minute hard cap. The failure mode is ugly: Lambda timeout kills an in-flight parse, the SQS message goes back to the queue, and the next attempt re-parses everything from scratch.

4. Retry semantics are wrong

Parsing is idempotent (same PDF → same JSON) and benefits from aggressive retries. Agent work is not idempotent (sending a reply twice is bad) and needs careful dedup. Bundling them into one handler means we tune SQS visibility + retry policy for the worse-behaved half.

5. Observability is muddled

When a report parse fails, the failure surfaces as a generic Clara error because there's no dedicated span or CloudWatch log group. Operators can't tell at a glance whether the rent-roll parser is unhealthy.

Decision

Extract attachment parsing into a dedicated Lambda — agents/attachment-parser — and wire Clara to it via a scatter-gather pattern over SQS:

┌──────────────┐    email            ┌──────────────────────┐
│  SES → SQS   │───────────────────► │   Clara Lambda       │
└──────────────┘                     │ (propflow-inbound)   │
                                     └──────────┬───────────┘
                                                │
                          1. raw email → S3     │
                          2. publish parse task │ (per attachment)
                                                ▼
                                     ┌──────────────────────┐
                                     │  attachment-parse    │
                                     │      SQS queue       │
                                     └──────────┬───────────┘
                                                │
                                                ▼
                                     ┌──────────────────────┐
                                     │ attachment-parser    │
                                     │ Lambda (fan-out)     │
                                     └──────────┬───────────┘
                                                │
                                   parsedAttachments → S3
                                                │
                                                ▼
                                     ┌──────────────────────┐
                                     │  inbound SQS (back)  │
                                     │  email-enriched msg  │
                                     └──────────┬───────────┘
                                                │
                                                ▼
                                     ┌──────────────────────┐
                                     │   Clara Lambda       │
                                     │ (agent loop runs)    │
                                     └──────────────────────┘

Concretely:

Consequences

Positive

Negative / costs

Rollback

The orchestrator is env-var-gated. To roll back: unset any of SQS_ATTACHMENT_PARSE_QUEUE_URL / SQS_INBOUND_QUEUE_URL / S3_ATTACHMENT_BUCKET on the Clara Lambda config, and it reverts to inline parsing without a code deploy.

Migration plan (phases)

  1. Phase 8.2 (this commit): skeleton Lambda + ADR. Lambda builds and smoke-invokes with _smoke: true fixtures but does not yet receive real traffic.

  2. Phase 8.3: Clara gains attachment-orchestrator.ts helpers + email-enriched channel. Still env-var-gated OFF in prod.

  3. Phase 8.4 (infra): create the SQS queue + S3 prefix + IAM permissions. Flip env vars on in stage/preview first.

  4. Phase 8.5: flip env vars on in prod. Monitor. Once stable, delete the inline parser imports from Clara's bundle and mark pdf-parse/xlsx/papaparse as external in Clara's build.ts.

  5. Phase 8.6 (optional): refactor Vercel admin routes to call the attachment-parser Lambda Function URL. Only if the bundle-size savings justify the added latency.

Alternatives considered