PropFlow.ai PropFlow.ai Docs
Accepted / Implemented — **option A (serialize per account)** shipped; **option B** deferred until volume justifies it; **option C rejected** (Fede: no second AppFolio login).markdown source

ADR-0067: AppFolio session concurrency — serialize vs. session-refresh-lock for parallel renewal prep

Context

On 2026-06-23, the first real autonomous renewal batch for Camellia fired three prep jobs within ~6 seconds (Mia, Wanda, Verity). The first (Mia) succeeded; the second and third collided on the shared AppFolio session and failed their first attempt with:

session bootstrap failed: REFRESH_DID_NOT_TAKE: ...still returned HTTP 401 after a full login refresh — session could not be re-established.

They only succeeded on later SQS retries once spaced out in time. One side effect (Verity) was an orphaned AppFolio offer and a spurious PM escalation.

Crucial reframing — the collision is no longer a correctness problem. Two fixes shipped 2026-06-24 already neutralize the harm:

So a collision now simply costs latency (a batch member waits for a 150s SQS redelivery and succeeds on attempt 2–3). It no longer corrupts state or pages a human. This ADR is therefore a performance/throughput decision, not a fire — which materially lowers the appetite for a high-risk change.

Current architecture

The AppFolio browser-automation path is shared by all agent jobs — renewals, work orders, work-order enrichment, cancellations:

  1. PropFlow publishes a job to propflow-agent-jobs.fifo (src/lib/integrations/sqs/agent-jobs.ts). Renewal prep uses MessageGroupId = appfolio.renewal.send:<leaseId> — per-lease, so different leases land in different FIFO groups and are delivered concurrently to separate Lambda invocations (propflow-agent-runtime-prod, reserved concurrency 25).
  2. Each Lambda POSTs to the single appfolio-browser-agent Vercel deployment.
  3. That deployment uses one Browserbase persistent context for all jpco work (single BROWSERBASE_CONTEXT_ID; AppfolioAccountConfig.browserbase.contextId is a scalar, not an array). The context holds AppFolio's Rails _session cookie and device-trust cookie.
  4. AppFolio login = email/password plus SMS-based MFA (code delivered to one Twilio number, +18442853526). The existing MFA lock (mfa-locks.ts, PK MFA_LOCK#sha256(orgId+email), PR #181) serializes only the MFA-code-submission step — not the full session bootstrap. Device trust does not persist in the Browserbase context (verified 2026-05-26), so every session lapse forces a full MFA re-login.

The collision, precisely

When the shared _session cookie is stale, N concurrent jobs all detect 401 and all enter the login/refresh path at once. One wins and writes fresh cookies into the shared context; the others' in-flight logins are invalidated by AppFolio (a new login for the same device/credential kills the competing sessions) → their post-login re-probe still 401s → REFRESH_DID_NOT_TAKE. The L4 retry then fires /api/cron/refresh-if-needed?force=1 for the losers, which races again on the same shared context. They recover only on later SQS redeliveries, once the winner has finished and released the session. The MFA lock does not prevent this — it guards code submission, not the bootstrap/refresh sequence.

This is not renewal-specific: any two concurrent AppFolio browser operations (e.g. a renewal prep + a work-order write) on a stale session can collide the same way. The renewal batch merely surfaced a latent system-wide property.

Options

Change the FIFO MessageGroupId for AppFolio browser jobs from per-lease to per-account (e.g. appfolio:<accountId>). SQS FIFO guarantees in-order, one-at-a-time delivery within a group, so only one AppFolio browser op runs per account at a time — the collision cannot occur.

Option B — Session-refresh lock (the "Phase 1" from the concurrency investigation)

Add a DDB SESSION_REFRESH_LOCK#<account> that serializes the entire session-bootstrap-or-refresh (not just MFA submission), mirroring mfa-locks.ts. On a warm session the lock is never taken → N jobs run in parallel. On a stale session exactly one job re-logs-in while the others receive 503 SESSION_REFRESH_LOCK_HELD and park via SQS ChangeMessageVisibility, then retry onto the now-warm session (no second MFA).

Option C — Multi-credential session pool (rejected)

N independent Browserbase contexts, one per AppFolio login (e.g. fede@ + gera@), leased per job — true N-way parallelism even for writes. Rejected by Fede: no second AppFolio login. Documented for completeness; revisit only if a second service identity ever becomes acceptable.

Adopt Option A (serialize per account) now. Rationale:

  1. The collision is no longer a correctness issue (#2592 + #2595). What remains is latency, and at current volume serialization adds none that matters.
  2. A is one low-risk line vs. B's broad, cross-service surgery on the session path every AppFolio op shares.
  3. It directly satisfies the "if too complex, stagger them" guidance.

Defer Option B until AppFolio browser-op volume (renewals plus work orders, across more than one property) actually makes per-account serialization a throughput bottleneck — and, before building it, first answer the single-active-session-per-credential question (a cheap experiment: two concurrent warm-session writes; observe whether one is invalidated). If AppFolio enforces single-session, B's value is capped at parallel reads and C (rejected) would be the only true write-parallel path — which further argues for staying on A.

Reject Option C per the no-second-login constraint.

Consequences