RCA: AppFolio robot login lockout, Sep 9–11 2026

Why the browser robot that writes into AppFolio locked itself out twice in two days, and what changes so it doesn't happen a third time.

Written 2026-09-11 morning from three read-only investigations (change history, live system evidence, and a walkthrough of how the pieces fit together). All times America/Denver, 12-hour clock. Every number below traces to a database query, a Sentry error record, a Browserbase session log, a Twilio record, a code/deploy record, or a session transcript. Anything marked OPEN is still being run down.

1. What a customer would notice

Since 4:53 PM on Sep 10 (JP&Co and Situs Group) and 3:58 PM on Sep 10 (Western Slope), the robot that writes into AppFolio can't sign in. Every write it attempts gets refused.

No tenant, prospect, or vendor was affected. This was checked directly — a full scan of 6,941 automation-run records since the lockout, logs across five backend jobs, all 3,022 work orders, and all three retry queues (empty). Test-property and stress-test traffic was excluded from every count.

Zero real-property customer-facing breakage. No real work order went unsynced (the only two stuck rows are Willows test-property stress rows). No real message to a prospect failed (the six failures were test stress addresses; the actual guest-card lookups run on a different, unaffected system and succeeded 453 out of 453 times). Nothing was sent wrong, withheld wrongly, or partially written.
Vendor-contact refresh at Camellia — an internal cache of vendor phone/email, not customer-facing — has fallen behind: failures climbing 318 → 393 → 425 of 819 vendors, 282 unreached. Worst case, a vendor's number that changed in AppFolio during the outage is stale in our cache. No evidence that happened. This cache self-heals on its own next successful run, oldest entries first.
Work-order enrichment at Camellia — an internal detail card (which vendor is on file, time slots, permission to enter) — is stale for 21 work orders. This is a staff-facing display cache only. It also self-heals on the next successful run.
Nightly PMS sync canary and turnover check: red three nights running.
Willows test-bench unit TEST-102 wasn't auto-released (test property only).
Cost: about 135 short, failing browser sessions per hour since 4:00 PM Sep 10 — nine times the normal rate, all doomed probes.

Not affected at all: Clara's answers about unit availability (that comes from a different, read-only feed that ran 488 straight successful pulls at Camellia through this whole window), rent roll data, and renewals data. Nothing needs a manual redo, and nothing is lost.

2. Two separate outages, not one

Outage A — Sep 9, 3:06 PM to about 5:00 PM. Fixed that night.

A real sign-in failure. The robot's saved browser profile got stuck in a redirect loop with AppFolio's login page — 20 attempts in 37 seconds, never even reaching a password box. Chasing it created 21 throwaway browser profiles that day, and every new profile costs a fresh text-message security code: 23 AppFolio codes went to the fede@ number in 53 minutes, and 28 went to the Western Slope number the night before.

3:06 PM — a new safety limit on logins (installed at 1:14 PM that same day) correctly trips for the fede@ login (shared by JP&Co and Situs Group), locking it for 30 minutes.
3:07 PM — a person reviewing the situation sees the lock, and deliberately extends it to 12 hours so nothing else would try to log in while things were sorted out. This was a judgment call, not a bug.
~4:50–5:00 PM — restore: the browser profile is rotated, the lock is cleared 10 hours early, one carefully watched login is run (it cost 2 login-budget units instead of the usual 1), both the staff page and the underlying data connection are proven working, and a real work order is created and cancelled at the Willows test property to confirm writes work.
10:09 PM and 10:40 PM — all three accounts (JP&Co, Situs Group, Western Slope) confirmed logged in and healthy. Confirmed again at 10:02 AM and 11:18 AM the next day.
Residue: the fede@ login had already used 6 of its 8 allowed logins for the day by 11:18 AM on Sep 10 — only 2 logins of headroom left going into the afternoon.

Outage B — Sep 10, 3:58 PM onward. Still live.

A different cause: a loop that keeps re-triggering itself without any actual login happening inside it.

3. The mechanism — verified

Confirmed from the code, from Browserbase's own session recordings, and from Sentry's error trail. Roughly every 6 hours, when the safety lock expires, a routine "keep the session alive" check gets one attempt through — and inside a single already-open browser window, in about 8 seconds, this happens:

The keepalive check opens a browser and loads AppFolio's home page. It loads fine — the saved browser cookies are good, the basic access check passes, and a fresh set of session cookies is collected.
The robot checks those cookies against two separate doors: the staff website, and the underlying data API. The staff website says yes. The data API says no (HTTP 401). Verdict: the session is dead. It logs "driving a login refresh."
It calls the login routine — but never actually navigates away from the page it's already on. The login routine's first move is to ask "what's on screen right now?" The answer is the normal dashboard, so it reports back "already logged in" in about 2 seconds. No login form was shown, no password was typed, no security-code text was sent.
The code ignores that "already logged in" answer, re-collects the exact same cookies, and checks them again. Same failure on the data API. It then raises an error, REFRESH_DID_NOT_TAKE, with the message "Drove login successfully but the session it produced is dead (api door 401)." That message is false — nothing was actually driven through a login.
That error lands inside a piece of code (moved there by a change deployed at 11:05 AM Sep 10) that treats it as a bad-credential failure. This is the second such failure counted in 24 hours, which trips a stricter 6-hour lockout. The keepalive check tries again when that lockout expires — and repeats the same failed sequence. Forever, until someone intervenes.

Evidence backing this:

The four counted failures per login in the last 24 hours land exactly on a 6-hour schedule (10:32 PM, 4:32 AM, 10:32 AM for the clara@ login; 11:25 PM, 5:25 AM, 11:25 AM for fede@).
The Browserbase sessions during those windows ran 12–16 seconds versus a normal 6–7 second baseline — and none of them were actually a real login flow.
Zero AppFolio security-code texts have arrived on either phone number since Sep 9 (5:20 PM for fede@, 3:56 PM for clara@) — real logins would generate one every time.
The storage system meant to record login-failure details is empty, because this particular kind of error never reaches it.
The browser profiles haven't been reset since Sep 9/10, which rules out a separate "stuck in a login loop" detector as the cause.

Both logins failed in parallel because they share the same code path, not because they share a password. First trip: Western Slope at 3:58:24 PM, triggered by the keepalive check. JP&Co/Situs Group: 4:53 PM, triggered by a real incoming message about a prospect that needed a guest-card write.

Things we built that made it worse: our own calling code treats REFRESH_DID_NOT_TAKE as worth retrying, so it forces another login attempt. The robot's refusal comes back as an HTTP 503 error, which the message-queue side then redelivers up to four more times. And the keepalive check itself was widened on Sep 9 at 3:53 PM from covering just the JP&Co login to covering all three accounts every 2 minutes — so now three customer accounts sharing two logins are all hammering the same login-budgets constantly.

4. Root cause — AppFolio sessions have a hard 24-hour lifetime

Found. The 24-hour cap itself is inferred at high confidence; everything else below is directly verified.

Signing into AppFolio sets three cookies: one company-wide sign-on cookie good for about 29 days, one "remember this device" cookie good for about 30 days, and one short-lived per-database session cookie. Our keepalive check pings the data API every two minutes, which resets that session cookie's idle clock (about 3 minutes) — but it cannot reset its separate absolute clock. Once that absolute clock runs out, the company-wide sign-on cookie is still perfectly valid, so the staff dashboard still loads and the basic access check still passes — while the data API flatly refuses it. That mismatch is what section 3 calls "hybrid-stale." We proved this exact behavior once before, on 2026-05-25, and it's written into our own troubleshooting notes: a full password-and-code login can restore the data API door.

The timing lines up almost to the minute. The last real logins were Sep 9 at 3:56 PM (clara@ login) and 5:20 PM (fede@ login). The break happened Sep 10 at 3:58 PM — 24 hours and 2 minutes after the clara@ login. All three accounts were separately confirmed working on both doors at 11:13 AM and 11:18 AM on Sep 10, 19 hours in. Our own troubleshooting notes had already clocked the longest a session had ever stayed warm at about 23.3 hours, and had an open question asking whether there's a hard ceiling somewhere past 23 hours. This incident answers that question: yes, 24 hours.

Why this never showed up before, and why all three accounts broke at once: until Sep 8 at 10:11 PM, every write ran a full fresh login and started from a throwaway browser profile each time — so no session ever lived long enough to hit the 24-hour wall. A change that night switched to reusing a saved, already-logged-in browser session instead of logging in fresh every time. The next afternoon, a second change made that saved browser profile persistent and put all three customer accounts on the same indefinitely-reused keepalive. The first 24-hour boundary after both of those changes landed was Sep 10 at about 3:56 PM — which is exactly when it broke.

Why it can't fix itself: the login routine decides a session is fine purely from seeing the normal dashboard on screen — the exact same shortcut that was deliberately removed from a different check back in May for producing false positives. The code calling it throws that "already logged in" answer away, no password is ever actually typed, and that non-attempt then gets counted as if it were a failed login with bad credentials.

Ruled out: duplicate same-name cookies colliding across the different AppFolio sites we log into (the mechanism for that exists in our cookie-handling code, but the same saved session worked fine for 19 hours on the same path first, and a naming collision would fail immediately, not exactly at hour 24); a cookie missing a security flag it needs (same reasoning, and it would point the opposite direction from what we saw on Sep 9); and any code deploy or scheduled job at 3:58 PM (there wasn't one — the last deploy was 4.7 hours earlier and had already passed its own after-the-fact proof check). A previously-logged mistake about "a fresh login isn't the fix" turns out to be about the staff-webpage door, not the data-API door that's actually failing here — there's no prior case of a truly fresh login failing the data-API door.

What we still can't see: there's no way today to just look up the stored session's cookie names, which sites they're scoped to, and when each one expires — the tool that touches them only knows their names well enough to hide them in logs. Before the next real recovery login, we need to capture that information before and after, plus confirm directly that a password was actually typed and that the work-orders endpoint of the API comes back healthy afterward.

5. Why we broke it — what our process missed

Fifteen changes to the browser-robot code shipped in 46 hours (Sep 8 afternoon through Sep 10 lunchtime). Fourteen of them touched login, sessions, or cookies; nine touched the same three files. Almost every individual change fixed a real bug. Stacked on top of each other with no time to let any one of them prove itself, they produced a system that shuts itself off.

The test for the Sep 10, 11:05 AM change checked the wrong thing. That change altered what counts as a bad-credential failure, and its proof — passing 8 minutes after it went live — only checked that sessions that were already working stayed working. The exact failure path it rewrote was never actually tested.
A known problem got mislabeled as a bad password. Our troubleshooting notes recorded on Sep 7 that a fresh login can still fail this same door. On Sep 10, the 11:05 AM change turned that exact known condition into "the password is broken." Nobody connected the two.
Fixes chased each other in a circle. One change added a strict rule (any check that isn't a clean pass counts as dead). The next change fixed a bug where logins were being double-counted against the daily budget. The next change fixed the first change's classifier — but in doing so, moved the check to a place where it also undid the second change's counting fix. Each change fixed the side effect of the one before it, and the end result was stricter counting applied to a situation a perfectly healthy session can produce on its own.
Our own rollout rule wasn't followed here. The new login-limit change went live to all three real customer logins the same afternoon it was written. The keepalive-widening change went live to two customer accounts the same day it was written. The rule we use elsewhere — ship it turned off, test it on a safe test property, only a human turns it on for real customers — has no equivalent in this part of the codebase.
The daily login budget was almost gone before the day even started. 6 of the 8 allowed daily logins were already used up by 11:18 AM on Sep 10, left over from the previous day's incident. It only took two more false failures to run out.
The alarms didn't say why. The system records that a lockout happened, but never records the actual underlying error; there's no permanent log for this class of failure; the snapshot system that's supposed to capture failure details doesn't cover this error type. Three straight nights of red alerts got automated summaries ("self-heals," "escalating to Fede," "handing to the operator") because there was nothing concrete anyone could safely act on.
Nobody was actually watching the red alerts. Detection itself worked fine — the nightly canary check was red three nights running, the turnover check was red three nights running, and Sentry flagged it within minutes each time. No person was actually on the hook to look at it and act.
An unrelated login slipped through uncounted. On the evening of Sep 9, a separate, unrelated task drove logins from someone's local machine using the same already-compromised browser profile, bypassing both the lock and the budget entirely.

6. Timeline (Denver time)

When	What
Sep 8, 2:43 PM	Western Slope credentials added to production
Sep 9, 1:14 PM	Login budget + lockout rule goes live (3/hour, 8/day, 30-min lock then 6-hour lock)
Sep 9, 1:28 PM	Change requiring an explicit account on every call goes live; JP&Co fallback removed
Sep 9, 3:06 PM	Outage A: lockout trips for fede@ login (30 min)
Sep 9, 3:07 PM	A person deliberately extends the lock to 12 hours
Sep 9, 3:53 PM	Keepalive check widened to warm all three accounts every 2 minutes; browser profile kept between runs
Sep 9, 4:15 PM	Staff-page check added to startup, keepalive, and warm-session logic; login-loop detector added
Sep 9, 4:19 PM	Escalation alert observed, 4 minutes after the above change
Sep 9, ~4:50–5:00 PM	Restore: browser profile rotated, lockout cleared, one watched login run, real write proven at Willows test property
Sep 9, 5:20 PM	Last AppFolio security-code text ever received for fede@
Sep 9, 9:52 PM	(Separate system) sync dispatch changed to run per-connection: two workers hitting one AppFolio account; rate-limit errors jump from 30/day to 976/day
Sep 9, 9:53 PM	Fix goes live: one login attempt now counts as one budget unit, not double-counted
Sep 9, 10:09 / 10:40 PM	All three accounts confirmed warm, no lockouts active
Sep 10, 10:02 AM	Health check: warm, not locked, 6 of 8 daily logins already spent
Sep 10, 11:05 AM	Change goes live: every login must now prove itself at both doors; a dead verdict counts as a credential failure; the once-per-flow counting fix from the night before is removed
Sep 10, 11:15 AM	Next change goes live (current production version)
Sep 10, 11:18 AM	Post-change proof check: all three accounts warm, zero lockouts, fede@ still at 6 of 8
Sep 10, 3:58 PM	Outage B begins: Western Slope's saved session fails the data API door; falsely counted as a failed login; lockout trips
Sep 10, 4:00 PM	Browser session rate jumps from ~15/hour to ~135/hour and stays there
Sep 10, 4:32 PM	Western Slope escalates to the 6-hour lockout
Sep 10, 4:53 PM	JP&Co session dies mid guest-card message; fede@ lockout trips
Sep 10, 5:24 PM	JP&Co/Situs Group escalates to the 6-hour lockout
Sep 10, 11:11 PM	(Separate system) fix goes live: one worker per AppFolio account
Sep 11, 1:51 AM	Turnover check alert: third red night in a row
Sep 11, 3:30–7:46 AM	Four more alerts; automated summaries only, no human action
Sep 11, 10:32 / 11:25 AM	Next scheduled lockout expiries — the loop re-triggers again unless the fix lands first

7. What changes — fix chain

A surgical chain of small changes to the robot's codebase, each with its own test that fails before the fix and passes after:

merged Never trust a dashboard landing as proof of login: if the login routine reports "already logged in" while the data-API door is still dead, force one real, full credential login (sign out, then log back in through the actual form) and re-check both doors. (#356, #357)
merged A login attempt that never typed a password is not a credential failure: give it its own error code, LOGIN_NOT_DRIVEN, exempt it from counting against the lockout, and remove the false "Drove login successfully" message. (#356)
merged Re-mint proactively: a nightly keepalive re-mints each account's session 2–5 AM local before the ~24-hour ceiling, kill switch SESSION_REMINT_ENABLED. (#358)
merged Actually record the real login error text in Sentry, and name the failure that tripped the breaker alert. (#359)
merged On our calling side (the main app, not the robot): stop treating REFRESH_DID_NOT_TAKE, the new LOGIN_NOT_DRIVEN, and the 503 refusal as things worth automatically retrying into an active lockout — login-breaker refusals are terminal, no forced refresh, no retry, queue does not redeliver. (#7794)
merged Alerting: one alert per customer organization (Fede's call, 2026-09-11), never one per job or per property. One outage-alarm row per organization with per-job pages suppressed, and one Slack message per organization edited in place — both dark behind PMS_OUTAGE_ALARM. (#7796, #7797)
merged Rotate the browser profile when the login lands in a redirect loop, so a stale profile doesn't strand an account. (#360)
merged A replay gate over eight real login landings, runs on every PR touching the sign-in layer — making it a required check is a branch-protection click Fede owns. (#361)
merged A further PR making the redirect-loop landing not count as a credential failure. (#363)

See section 8 for the full merge record and per-account recovery evidence.

Prevention going forward:

A required test — using a fresh login against the Willows test property, exercising both the failure and the recovery path — before any change to login/session code can merge.
One alarm per cause, not per consumer (Fede, 2026-09-11: "why am I getting pinged five times?"). Right now, five jobs failing on one locked-out login each raise their own separate alarm on every scheduled run, while the actual cause only ever reaches Sentry as a low-priority warning. Going forward, alerts group per customer organization, never per job or per property, and downstream jobs attach to that organization's one alarm instead of raising their own. Test-bench and internal-cache failures stay warnings, not pages.
The three-step rollout (ship dark, test at the bench, a human turns it on) applied to the browser-robot codebase, the same as everywhere else.
A seven-day watch after the fix ships: a daily report covering both logins, budget counts, lockout state, security-code text traffic, and browser-session rate.

8. Recovery and fixes shipped (2026-09-11)

Robot repo — merged and deployed to production

merged #356 a login the driver never drove is not a credential failure (8:28 AM)
merged #357 force one real credential login when the API door is dead (8:47 AM)
merged #358 keepalive re-mints the session nightly 2–5 AM local before the ~24h ceiling, kill switch SESSION_REMINT_ENABLED (9:14 AM)
merged #359 the breaker alert names the failure that tripped it
merged #360 rotate the browser profile when the login lands in a redirect loop
merged #361 replay gate over eight real login landings, runs on every PR touching the sign-in layer (making it a required check is a branch-protection click Fede owns)

merged The further PR making the redirect-loop landing not count as a credential failure — #363, production deploy 6397822869.

Main repo — merged

merged #7794 login-breaker refusals are terminal (no forced refresh, no retry, queue does not redeliver)
merged #7796 one outage alarm row per organization with per-job pages suppressed (dark, PMS_OUTAGE_ALARM=on to arm)
merged #7797 one Slack message per organization, edited in place (dark, same flag)

Recovery (Fede's go — "so unlock it")

JP&Co / Camellia — attempt 1 at 9:27 AM failed with ERR_TOO_MANY_REDIRECTS before the login form (stale browser profile, same family as Sep 9); attempt 2 at 9:37 AM after rotating the profile: real login, 89 seconds, 3,057-byte jar, both doors warm at 9:39 AM, still warm at 10:40 AM with no further logins.

Western Slope — same redirect loop on attempt 1, profile rotated, attempt 2 refused because its pause had re-armed; stopped at two attempts; paused until 3:42 PM, expected to self-heal on the next allowed attempt now that #360 is live.

Situs Group — not attempted: it shares JP&Co's login, whose daily allowance reached 8 of 8; expected to self-heal after the window rolls (~3:00 AM Sep 12).

Neither Situs nor Western Slope has live customer traffic.

Two internal caches at Camellia self-heal on their next runs: vendor-contact refresh at 1:30 PM Denver, work-order enrichment at 3:00 AM Denver.

Fede rulings recorded: alerts are grouped per customer organization, never per job or property; no customer-outage page.