Overnight Report — Aug 15

What ran while you slept, from the AWS credit review through a live incident. Everything below is verified against the live account, the repo, and the logs unless it says otherwise. Written 2026-08-15, ~9:50am Denver.

⚠️ Needs you first: Clara has been answering with the fallback line since 9:40pm last night

Since 9:40pm Thursday (Denver) — about twelve hours and still going as of 9:21am — every text or email a prospect sends gets the canned "Thanks — our team has it, someone will get back to you" instead of a real Clara reply. Cause: our production AI account is being told "this request would exceed your account's rate limit" on every call.

I checked every one of our own systems that talks to the AI: none of them were calling it during the outage (the nightly jobs run keyless or after the fact; no eval ran on the paid key; the review bots run on the subscription). So this is almost certainly the paid key's own account limit or spend cap on the Anthropic side, not our traffic. Only the Anthropic Console can confirm or lift it.

What to do: Anthropic Console → Usage / Limits for the production workspace key. If there's a monthly spend cap or a lowered rate tier, raise it. Replies resume on their own the moment the limit clears (nothing is stuck on our side — the one wedged test message already aged out).

Small silver lining: the message classifier handled the limit gracefully (kept going with a default), so the "graceful degradation" pattern exists — the main reply loop just doesn't use it yet. Ticket filed below.

What shipped (all merged, all deployed)

Area	What changed	Proof
Backups & audit readiness from the AWS credit plan	Daily backups (35-day keep) + a monthly copy stored in Oregon (1-year keep) for all six production tables; live-restore turned on for the three tables that lacked it; the infrastructure change recorder (AWS Config), continuous security checks (Security Hub), and the external-access analyzer are all on.	First on-demand backup of the main table (1.5 GB) completed; Config recorder confirmed recording; alarms verified.
AppFolio sync stops dying	The sync that mirrors AppFolio every minute was being killed at the 15-minute limit — not once, 112 times (Aug 9–11). Root cause: it ran pinned at 100% of its memory for 11 straight days, crawled, and got killed mid-property. Five fixes merged: a time budget so a run always exits cleanly (and reports what it skipped), memory doubled to 2 GB with a "getting full" alarm and the deploy script fixed so a manual deploy can't silently shrink it back, a retry cap on bad ticks, and the read pattern that was eating the memory (one query per tenant per tick → one per property).	Alarm live and OK; memory 2 GB confirmed; every fix has a regression test proven to fail on the old code. Cost: ~+$21/month.
The text loop gets faster	Your 51s and 33s test turns were traced step by step: ~20s of every turn was spent loading every conversation the property ever had, messages included (1,770 on the Willows; Camellia is at 786 and growing) just to find your one thread — twice. That's now a direct lookup. Separately, on a first-contact text, ~11s of prospect-record bookkeeping sat in front of the reply; those writes now happen right after the reply goes out (with guards so a serverless freeze can't drop them).	Direct-lookup fix live since 11:32pm. Bookkeeping fix approved, merging. On the test bench the bookkeeping went from 11.3s → 0.76s. Full before/after on real texts is blocked by the outage above — the measuring harness is built and ready (see below).
AppFolio verification-code storm	You saw 16 codes in an hour; the real number was ~740 in two weeks (150 on Aug 14). Culprit: the seven "is the lease countersigned yet?" pollers (one per pending Camellia lease) each did a full AppFolio login with a fresh code every hour just to learn "not yet." Fix merged: they now ask a cheap read-only feed first (no login) and only do the real fetch when it says the lease is signed — with a hard rule that a real check still happens at least weekly so a stale row can never silently strand a notice.	Renewal worker with the fix rolled out at 9:37am. 120 codes today up to 9:27am; expect a sharp drop from here. Nothing was terminated — the pollers keep running, just cheaply. Caveat: the cheap feed only covers "waiting on the PM's signature"; a lease waiting on the tenant's signature still does the full walk once a day.

In plain terms

Four things that were quietly wrong got found and fixed in one night: backups didn't exist, the AppFolio sync was starving itself, Clara was re-reading the whole building's history on every text, and a background job was burning hundreds of login codes. All fixed, all with proof. The one thing I couldn't fix is the AI account limit — that's a console setting only you can reach.

The AI-limit outage, timeline

9:40pm — first "rate limit" error on a Clara reply. Your own Willows test at 8:45pm was before this and worked (that's the 51s / 33s data).
9:44pm–10:20pm — every attempt fails the same way; the one stuck message was my test number's, redriving every 6 minutes until it aged out. Real senders: none affected in that window besides the fallback text.
10:00pm–midnight — I audited every consumer of the paid key (all background jobs, the renewal worker, every web route, every scheduled workflow, the eval rules). None were calling it. No policy breach.
9:21am — still failing on a fresh test message. Ongoing at time of writing.

What was proven, and what wasn't

Claim	Status	How
Backups, Config, Security Hub, live-restore on	Verified	Live account queries after each change
Sync memory pinned 100% for 11 days → 112 kills	Verified	CloudWatch per-day peak-memory + timeout counts
Sync memory pressure was allocation, not a leak	Verified	Per-container growth curves plateau at ~1,000 MB for hours; a leak would climb until crash
20s/turn spent on the whole-property read	Verified	Lambda REPORT + agent trace on your two live turns; partition size counted
Prospect bookkeeping 11.3s → 0.76s	Bench only	Real prospect mint on the test property; not yet measured on a real prod text
Text loop end-to-end after the fixes	Not measured	Every harness run hit the outage. Indicative: on a failed 9:21am turn the pre-Clara phase took ~5.6s vs ~20s before
Verification codes drop after the poller fix	Pending	Worker rolled out 9:37am; check tomorrow's count vs 120–150/day
Outage cause is the key's own account limit	Inferred	All internal consumers ruled out; console not reachable from here

Incidents I caused or found, and cleaned up

The email stress harness is broken, and its cleanup is dangerous. I ran it to measure the memory change; the five test emails never reached Clara's inbox (delivery, not Clara), so no data. Worse, its cleanup step purged the shared production inbound queue and deleted a real-looking Willows prospect named "Fede Chapa (Chapa Cleaners)" because it matched by name. Nothing tenant-facing was in the queue at that moment (verified), but that's luck, not design. Ticket filed.
The AppFolio sync stores a Temporal API key in plain text in its environment — readable by anyone with AWS read access. Should live in Secrets Manager and be rotated. Ticket filed.
The old AWS credit doc now carries a note that the $5,000 credit isn't visible in billing yet, and the Aug 12 AWS email was an invitation to apply, not an award — worth finishing that application if the credit matters.

Follow-ups (Trello, This Week)

Email stress harness: sends never reach Clara; cleanup must stop purging the shared queue
Temporal API key in plaintext in the sync Lambda → Secrets Manager + rotate
Clara's main reply loop should degrade gracefully on a rate limit, not send the canned fallback
AppFolio sync 15-minute timeout — the fixes above close this; card can move once tomorrow's run shows zero kills.

Still open for you

Anthropic Console — lift the production key's limit (top of this page).
AWS credit plan — your package pick is still pending on the credit doc; the "safety floor" pieces are already on regardless.
Once the AI limit clears, I'll run the text-loop before/after and give you real numbers (harness is merged-ready as a script-only PR).

Notes for the record

Auto-merge race, twice: the review bot approves the exact commit it read, and auto-merge lands the moment CI is green — a follow-up commit pushed seconds later gets stranded on a merged branch. Both times the stranded commits were recovered onto a fresh branch and merged; going forward, check the PR state before pushing follow-ups.
Accounts: the new trinity@propflowai.co subscription became active mid-session; it should be captured into the account rotation bank so it survives future switches.
Camellia benefits from the direct-lookup fix automatically (786 conversations, same code path) — no separate work.

Sources: live AWS CLI (account 194925971150), CloudWatch Logs Insights, Twilio message logs, the propflowai repo and merged PRs, DynamoDB reads on the test property. Bench numbers come from scripts/willows-gmail-dup-harness.ts run 2026-08-15 15:32Z. Times in Denver unless marked.