00 · Oct 2 review: bugs in the pull, and the renewal plan
Proposed — pending review. Fede asked 10/2: inspect the whole AppFolio pull, find the bugs from the September refactor, check what Gera has done, and plan renewal reads as one call for all. Read-only inspection of code, logs and the database on 10/2. Nothing built.
Bottom line. The refactor works: JP&Co and Western Slope data is fresh to within about a minute, and most reports are already one call per customer. Renewals are the exception. They skip the sync and read AppFolio building by building, and that one hourly check causes 68% of Western Slope's refusals. Separately, a cost bug since 9/19 rewrites every pulled record on every sync.
Western Slope today (06:00–17:15 UTC, 10/2)
| Who reads AppFolio | Calls | Refused | One call per customer? |
|---|---|---|---|
| Everyday sync: guest cards, applications, work orders (every minute) | 1,865 | 382 | Yes |
| Everyday sync: balances, leases, occupancy (every 5 min) | 1,411 | Yes, except 4 buildings (bug 4) | |
| Monthly income report (hourly) | 49 | No, 3 calls per building (bug 5) | |
| Listings | 42 | Yes | |
| Hourly renewal check (runs although renewals are off at Western Slope) | 306 | 827 | No. 3 reports per building, 89 buildings, fixed order. Reaches 6 of 89 an hour. |
The budget is 6 calls per 15 seconds per customer. Western Slope uses about 23% of it on average. The trouble is bursts: the minute sync, the 5-minute sync and the hourly renewal check land together, and every caller gives up after 3 seconds. AppFolio still answered "too many" 46 times today below our own limit; cause unknown.
Bugs found
| # | What is wrong | Evidence | Effect | Fix and owner |
|---|---|---|---|---|
| 1 | Every pulled guest card, application and tenant is rewritten on every sync, changed or not | Confirmed ~140,000 writes last hour; written = seen 100% | Database writes went from about 1M a day to 5–7.5M a day after 9/19 | Skip unchanged records. Our lane. No PR. |
| 2 | Hourly renewal check sweeps every building, renewals on or not, always in the same order | Confirmed 6 of 89 reached per run | 68% of Western Slope refusals; the same buildings never get checked | Only buildings with renewals on; one call per report (see options). Our lane. No PR. |
| 3 | The renewals sync has been parked since 9/12 | Confirmed no runs in 8 days | Offers staff send by hand in AppFolio are never recorded (Western Slope 189-251); tenant-signed detection from that path stopped. No alarm. | Un-park, hourly instead of its old 5-minute slot. Gera's call, on his tab since 10/1. |
| 4 | Four Western Slope buildings re-read the lease report every 5 minutes | Confirmed 421 calls on 191, 194, 400, 411 | ~1,150 wasted calls a day. A prospect waiting to become a tenant never promotes, so the check repeats forever. | Reuse the company-wide lease pull from the same sync. Our lane. No PR. |
| 5 | Monthly income report only counts as done when all 3 months succeed; one refusal and it retries every hour | Confirmed 29–74 of ~69 buildings finish a day | Hourly retries; the “6 AM” schedule in the config is dead | Mark done per month. Our lane. No PR. |
| 6 | A refused renewal read for a tenant with no history pages the property manager “needs your review”, once per lease | Code only no Western Slope tenant has a verdict yet | False pages to Cheryl on day one; the 65-day dedup means they never correct | A refused read retries tomorrow and never pages. Our lane. See Monday. |
| 7 | Refusals are logged as “unattributed”, so nobody can tell which job drains the budget | Confirmed | Slower diagnosis | Gera's open PR labels the sync jobs; the renewal check still needs it. |
| 8 | Situs has not synced for 72 hours | Confirmed paused by staff during the 9/29 storm | Situs data stale; no alarm on a paused customer | Unpause and add an alarm. Fede/Gera call. |
| 9 | Guest-card rewrite loop | Mostly fixed 1.9% of writes are real changes | Now small next to bug 1 | Open issue for the two-cards-one-prospect case. |
| 10 | About 12 sync copies run at once | Confirmed, harmless so far | Not tied to a failure | Watch. |
Renewal reads: options
Renewals read AppFolio in four places: the 8:00 payment-history check, the 8:30 “is an offer already out” check, the hourly renewal check, and a per-lease check for expired leases. The everyday sync already pulls almost everything company-wide and owns the budget. “One call for all” means renewals stop calling AppFolio themselves.
| Option | What changes | Western Slope renewal calls a day | Trade-off |
|---|---|---|---|
| 1. Patch the leaks | Bugs 1–6 fixed; renewals keep reading per building, only where renewals are on | ~1,600 (22 buildings × 3 × 24) | Fast, but the bursts stay and grow with every customer |
| 2. Renewals go company-wide themselves | Each renewal job makes one company-wide call per report per run | ~75 | Still a second reader competing with the sync at 8:00 and on the hour |
| 3. Sync pulls renewal data once; renewals read only stored data (recommended) | The sync pulls the renewal report hourly and payment history once before 8:00, company-wide; the renewal engine reads the stored copy | ~25 | The 8:30 “offer already out” check sees data up to an hour old instead of live. A hand-sent offer in that hour would be missed until the next run. |
Bugs 1–6 are fixes in every option and ship with no switch. One thing is unproven for options 2 and 3: whether AppFolio returns a whole company's 12-month payment history in one call (no code or log shows it). That needs one read-only test, off-peak on JP&Co's account, not Western Slope's.
Monday at Western Slope
Supervised mode holds Clara's texts, but not the “needs your review” page to the property manager. No Western Slope tenant has a payment-history verdict yet. If AppFolio refuses the 8:00 read for a building on Monday, every tenant there is held and Cheryl gets one email and one text per lease, and the 65-day dedup means it never corrects (bug 6). The 8:00 check only reads buildings that have a renewal policy, and the policy is set per building. So setting it only on the approved tenants' buildings keeps Monday to one read per approved building instead of 25–35 at once (estimate). That is a setup step, no code. Recommended on top: fix bug 6 before Monday so a refused read waits a day instead of paging. It is a bug fix in our lane, no switch.
Gera
- Did not build the refactor. Fede's sessions built it overnight 9/11–9/13; Gera's overlapping budget design was closed on Fede's call and one idea from it kept.
- Parked renewals, vendors and work orders on 9/12 to stop the storm. Turned work orders back on today. The reasons for parking renewals are gone: the budget is always on and the company-wide renewal code exists.
- Has an open PR today labelling every sync job against the budget. Useful; recommend it merges as is.
- Two decisions sit on his tab: un-park the renewals sync, and the sync copies running at once.
- Gap: nobody has work open on bugs 1, 2, 4, 5, 6 or 8, or on renewal reads as one call.
For Fede
- Monday: fix bug 6 before turn-on, plus policy only on approved buildings: yes or hold. Recommended yes.
- Renewal reads: option 3? Recommended.
- One read-only test call for company-wide payment history on JP&Co, off-peak: yes or hold.
- Fix bugs 1, 2, 4 and 5 as small PRs, no switch: yes or hold.
- Situs: unpause with Gera, plus an alarm when a customer stays paused.
01 · What happened (Sep 11)
Like asking someone who checks the mailbox once a minute to instead move an entire family's belongings into a new house, one box at a time, on foot.
- 25,000 requests to AppFolio in about three hours.
- 76% of that spent on every-minute jobs retrying "slow down" errors.
- Rent roll finished 46 of 69 properties; availability, 14 of 69.
- Still climbing at hour three, nowhere near done.
- Team paused the load by hand at 8:35 PM.
Proves: sync job logs, 5:15–8:35 PM, Sep 11.
Deeper → Appendix: the full measurement table
01b · Before and after, in API calls
Last night one truck made twenty-five thousand trips; this morning the same move rode in three.
- Old way: one call per property, per job, per timer — ~6,200/day for ONE property (math in appendix).
- JP&Co, 2 properties: ~12,400/day by that math — measured 13,910/day, close.
- Western Slope, 69 properties: ~429,000/day by that math — the 25,000-in-3-hours incident, throttled down.
- New way: one call per job for the WHOLE account — ~6,200/day, any portfolio size.
- Reduction: 2x for JP&Co, 69x for Western Slope.
Proves: schedules.json × property counts; Sep 11 logs, 13,910 req/day on jpco.
02 · Why: built for customers who already have data
A smoke detector is great at noticing a spark. It was never built to also be the fire hose for the first fire.
- Accreted job by job, one timer added at a time.
- The first pull was never designed as its own job.
- Sep 9 audit named a 3–5 customer ceiling in writing.
- That finding sat unactioned for two days.
- Completion records were retrofitted, not planned in.
Proves: written record — ADRs, PRs, session transcripts, Sep 9 audit.
Deeper → Appendix: the decision timeline
03 · The measurement that settles it
One truck for the whole move beats twenty-five thousand trips carrying one box each.
- A whole-portfolio call beats one call per property, always.
- Up to 5,000 rows come back per page, one request.
- About 7 requests per 15 seconds, per AppFolio login.
- Projected full first load: about 11 calls, not 25,000.
- Needs re-measuring once the old sync is fully paused.
Proves: bulk pilot, Sep 11 — rent roll and unit list, both clean.
Deeper → Appendix: the correction and caveats
03b · Architecture options, ranked
Recommended: option 1. Runner-up: 4, for the first load only, if stages ever branch. 3 is an additive later step, not a requirement.
1 · Lambda + DDB bucket
One Lambda, one bucket per account, gates the rate.
buys: no new services · costs: nothing
2 · EventBridge Scheduler
Per-account schedules replace the shared rule table.
buys: no-redeploy cadence · costs: unneeded console
3 · SQS FIFO
A queue decouples trigger from call.
buys: DLQ visibility · costs: new queues
4 · Step Functions (import)
Native per-stage retry, visible execution graph.
buys: native checkpoint · costs: 2nd orchestration model
5 · Fargate worker
Removes Lambda's 15-min ceiling; a fleet to run.
buys: no timeout · costs: new compute surface
| Option | New services | Isolation per key | First-load resume | Cadence change needs deploy? | Ops surface | Simplicity |
|---|---|---|---|---|---|---|
| 1 · EventBridge+Lambda+DDB | None | DDB bucket per account | Self-invoke + DDB checkpoint | Yes | One Lambda's logs + AutomationRun | 5/5 |
| 2 · EventBridge Scheduler | Scheduler groups/schedules | Same DDB bucket | Self-invoke + DDB checkpoint | No (API update) | + Scheduler console | 3/5 |
| 3 · SQS FIFO | Queue(s) + DLQ | DDB bucket (queue is not per-key) | Self-invoke + DDB checkpoint | Yes | + queue depth/DLQ dashboard | 3/5 |
| 4 · Step Functions | State machine | Same DDB bucket | Native state-machine checkpoint | Yes | + Step Functions console | 3/5 import, 2/5 refresh |
| 5 · Fargate worker | ECS/Fargate, ECR, IAM | In-memory, per task | Needs DDB checkpoint (memory lost on crash) | Yes, unless paired with Scheduler | + ECS task health | 2/5 |
Proves: architecture research, Sep 11 — steady-state audit: ~170k → under 18k requests/day.
03c · The engine, explained
Three speeds — right now, while you wait, end of day — one shared fuel gauge, and a switch for every gear that only Fede flips.
- Three speeds, defined once for everyone: act_now 1 min, answer_live 5 min, bookkeeping 1 hr.
- One shared budget per AppFolio account — every job spends from the same gauge.
- Each job's portfolio-wide call is its own dark switch, off by default, Fede-only to arm.
- A job can be PARKED (renewals, lease_states, work_orders, vendors) — survives deploys, needs a reason.
- New customers get a separate one-time import, residential properties only.
Proves: PRs #7966, #7911/#7924, #7923–#7953/#8126, #7965; residential ruling, Sep 12.
Deeper → Appendix: job states and file paths
04 · Onboarding, old vs new, in minutes
A moving crew that packs the kitchen before the garage and can pick up tomorrow, instead of one person carrying boxes at random until the building says slow down.
- Old way, Sep 11: 54/69 buildings failed, 146/234 units missing after 4h20m.
- New way: one-time 5-stage import, own budget, checkpointed, resumable.
- Willows bench, full chain: stage 1 7.6s, stage 2 11.0s (small test property).
- Stages 3–5 clean run: 2.8 minutes, 2,799 rows, 10 AppFolio calls.
- Full-portfolio result: see Ch. 07b — reconciled, with a known unit-count gap.
Proves: Sep 11 logs (54/69 failed, 146/234 units short); bench-first-load.md, run 2.
Deeper → Appendix: the five stages in detail
05 · What the customer sees
Like a package tracker: you see "out for delivery," not the truck's whole route.
- Starts as soon as the admin confirms which properties (default).
- Admin can close the laptop and walk away.
- Can sit dark for a few days before going live (Fede, Sep 9).
- One alarm per customer, only after a real failure.
- Never a flood of pages for one bad night.
Proves: Sep 9/10 rulings on alerts and stale-data handling.
Deeper → Appendix: the rulings list
06 · The everyday sync at 10x
One phone call to ask about every unit in a building beats calling once per unit.
- One call per customer per job, not per property.
- Seven jobs get a slower, still-workable cadence.
- One check removed outright; one folded into another.
- JP&Co already tops 1,000 rate-limit hits a day.
- 10x projection: ~170,000/day today's design vs ~14,000–18,000 proposed.
Proves: steady-state audit, Sep 11.
Deeper → Appendix: the per-job table
06b · JP&Co: fully on the portfolio engine
Turning every gas pump back on, one at a time, watching the gauge after each — until the whole station runs clean.
- JP&Co is fully on the portfolio engine: budget, occupancies, guest cards, balances, applications, lease states all armed; renewals stays parked; daily schedule went back on last.
- Balances errors: 8/hour before → zero next hour.
- Account-wide "slow down" replies: ~14/hour before → 7–9/hour after.
- No error or no-data alarm fired; one 11-minute warning cleared itself.
- Leftover "slow down" replies traced to test properties, fixed same day.
Proves: ops watch and prod logs, jpco account, Sep 13 (cutover-watch file).
07 · Decisions
Eight forks in the road: five settled, three still open.
| # | Decision | What's decided / default | Status |
|---|---|---|---|
| 1 | Build shape | Decided (Fede, Sep 11): EventBridge + sync Lambda, per-account budget in DynamoDB, portfolio-wide calls, first load as a checkpointed stage chain. Built. | Decided |
| 2 | Import start | Decided: on property confirmation. | Decided |
| 3 | Hand-off | Decided: after the whole import. | Decided |
| 4 | History depth | Decided: defaults per module, configurable. | Decided |
| 5 | Everyday cadences | Decided (Fede, Sep 12 AM): three global classes by report — act_now 1 min, answer_live 5 min, bookkeeping 1 hr. Declared in schedules.json. | Decided |
| 6 | Turn it on for JP&Co | Done, Sep 13: fully on the portfolio engine — budget, occupancies, guest cards, balances, applications, and lease states all armed, plus the daily schedule (Ch. 06b). Retiring the old per-job timers stays Fede's call. | Done |
| 7 | Reconcile with Gera's #7906 | Decided: #7906 closed; his declared-schedule-state idea shipped as PR #7965 (state survives deploys, mandatory reason). | Decided |
| 8 | Western Slope | In progress: Run 2 reconciled 69/69 buildings, 159/234 units (Ch. 07b); unit-mapping gap and financials-stage fixes in flight, sync pause lifted, everyday sync on the same engine. | In progress |
Row 6's last step, retiring the per-job timers, stays Fede's call.
Proves: Fede's decisions Sep 11–12; Sep 13 arming checkpoints.
Deeper → Appendix: history windows per module
07b · Western Slope re-import
Emptying a garage, then packing it again in order — and counting what's on the shelves before calling it done.
- Run 2 reset 2,498 rows, imported all 69 buildings, four stages clean: 1,049 guest cards, 191 applications.
- Reconciled 69/69 buildings, only 159/234 units — mapper drops a blank "Unit" name; fix in flight.
- Dashboard financials failed on throttling. Fixed 3:34 PM: one call per building, not ~1,400.
- Leasing write speed fixed 3:59 PM: 8 properties at once, ~8x faster.
- Pause lifted 2:25 PM; first hour clean — a few jobs still poll per building, under diagnosis.
Proves: reimport-result-2026-09-13.md (Run 2); unit-gap-diagnosis-2026-09-13.md.
08 · The road
Every new part of the plumbing gets tested with the water off before it's turned on for a real house.
- Six small PRs build the import, in strict order.
- Two more PRs bring the everyday sync to 10x.
- Every step ships dark, then gets tested at Willows.
- Then a read-only replay against real customer data.
- Nothing runs at a real property the same day it's built.
Proves: the standard rollout playbook — dark, tested, then Fede's call.
Deeper → Appendix: PR-by-PR proof plan
A1 · Appendix — full record
Every table, file path, and job name from the working record, for whoever builds this next.
Measurements by report (Sep 11)
| What | Result |
|---|---|
| Western Slope's real first load (one call per property) | 82 properties found, narrowed to 69 real ones; ~25,000 requests over 3 hours; 76% spent on 1-minute jobs retrying into "slow down" errors; only 2 of 16 jobs ever finished a pass |
| Bulk pilot — rent roll, whole portfolio, one call | 239 units returned, 5.2 seconds |
| Bulk pilot — unit list, whole portfolio, one call | 240 units returned, 1.5 seconds |
| Bulk pilot — next 3 calls attempted (tenants, vacancies, prospects) | all three rejected with "slow down" — likely the old sync competing for the same account's speed limit at the same time |
| Projected full first load, bulk design | ~11 total calls (one per report needed) — needs re-measuring clean, without the old sync running |
Before/after math, by the numbers
Old way = one AppFolio call per property, per job, on that job's own timer (schedules.json expression, unchanged by the cadence-class work). New way = one call per job, for the WHOLE account, regardless of property count (the portfolio arms in Ch. 03c/06b). Sum below is every Reports-API job in schedules.json at its declared cadence (excludes wo_enrichment/vendor_contact, which are L4 staff-portal sweeps, not Reports-API pulls).
| Job | Cadence | Calls/day, one property |
|---|---|---|
| work_orders | 1 min | 1,440 |
| lease_states | 1 min | 1,440 |
| renewals | 1 min | 1,440 |
| guest_card_messages | 2 min | 720 |
| occupancies | 5 min | 288 |
| rental_applications | 5 min | 288 |
| prospect_conversions | 5 min | 288 |
| balances | 15 min | 96 |
| guest_cards | 15 min | 96 |
| purchase_orders | 15 min | 96 |
| vendors | 1 hr | 24 |
| payments | 1/day | 1 |
| financials | 1/day | 1 |
| lease_states_history | 1/day | 1 |
| Total, per property, per day | 6,219 |
| Account | Properties | Old way (per-property × 6,219) | New way (portfolio, 1× 6,219) | Reduction |
|---|---|---|---|---|
| JP&Co (org_jpco) | 2 | ~12,438/day (theoretical) — measured 13,910/day, Sep 11 logs; 12% higher, self-stacking retries | ~6,219/day | ~2× |
| Western Slope | 69 | ~429,000/day (theoretical) — the 25,000-in-3-hours incident is that pace, throttled well below it | ~6,219/day | ~69× |
Source: lambda/appfolio-sync/schedules.json cadences, Sep 12 (renewals/lease_states/work_orders/vendors are PARKED as of Sep 12 but counted here at their declared rate — the "old way" this compares against is the design as built, not today's paused subset). Property counts: 2 JP&Co properties confirmed by a read-only prod audit, Sep 11 ~10:15 PM (supersedes an earlier, stale 25-property figure in the table below); 69 Western Slope properties from confirmed-property-ids.txt.
Steady-state (everyday) sync, today
| What | Result |
|---|---|
| JP&Co (25 properties, per the Sep 11 audit — later corrected to 2 real properties, see above) — rate-limit hits | ~1,000–1,200 real "slow down" responses per day, on a single, smaller account |
| Situs (94 properties) — rate-limit hits | close to zero — not proof it's safe at that size, just not loaded as hard yet |
| Occupancy check on a 69-property portfolio | ran 11 minutes, finished only 19 of 69 properties before time ran out |
| Wasted attempts per check, by job | occupancy 64% · tenant-balance 63% · renewals 45% · move-out 43% · applications 52% · prospect-card 54% |
Per-job recommendation and 10x projection
| Job | Today's speed | Calls/day today | Wasted | Recommendation |
|---|---|---|---|---|
| Maintenance tickets | every 1 min | ~3,200 | — | Slow to 3–5 min and once per customer |
| Move-out detection | every 1 min | ~19,500 | 43% | Keep 1 min, call once per customer |
| Renewals | every 1 min | ~19,500 | 45% | Slow to 5 min, once per customer |
| Occupancy | every 5 min | ~3,900 | 64% | Keep 5 min, fix first, once per customer |
| Rental applications | every 5 min | ~4,000 | 52% | Slow to 15 min, once per customer |
| Prospect cards | every 15 min | ~1,300 | 54% | Keep cadence, once per customer |
| Tenant balances | every 15 min | ~3,650 | 63% | Slow to 30 min, once per customer |
| Today (2 customers, ~119 props) | 10x, today's design (10 customers, ~940 props) | 10x, proposed design | |
|---|---|---|---|
| AppFolio requests/day | ~20,300 | ~170,000 | ~14,000–18,000 |
Timeline of decisions, with sources
| Date | Decision | Assumption it made |
|---|---|---|
| 04-16 | Read customer data via AppFolio's reporting tool; write changes through our own browser robot | Reads and writes on two separate systems, no shared bottleneck — never addressed a brand-new customer |
| 04-17 | Every scheduled check runs on one shared timer system (AWS EventBridge) | Checks are small and repeat forever, not a multi-step one-time job |
| ~07-xx | Each check's frequency reasoned about one job at a time | A customer already has data sitting there; a first load has nothing sitting there yet |
| 08-01 | Stop re-reading unchanged data on every check | Targets routine-checking cost, not a first-time load |
| 09-10 | Clara always answers with the last successful data; alert only after 6 hours; one worker per account | Root cause: a burst of "slow down" errors tripped a shared safety switch and silenced Clara for everyone |
| 09-11, evening | Every check runs for every customer regardless of features turned on | Triggered by the maintenance check failing 82 times for Western Slope, who doesn't use it |
| 09-11, ~8:40–8:55 PM | Fede, watching the load live: the first pull needs its own purpose-built system | "Our current design was 100% for when there's already data... checking every minute is completely wrong" for a new customer |
The one-line finding: no design document, PR, or planning session ever designed the first load as its own problem before tonight. The Sep 9 audit was the first to name the 3–5 customer ceiling in writing, and it sat unactioned for two days.
Rulings this design obeys
- Sep 9: every customer's checks fully separated from every other's; a login belongs to the company, never one person; onboarding is admin-confirmed properties, not automatic; AppFolio API access is turned on before setup begins.
- Sep 10: Clara always answers with the latest data however old; team alerted only after 6 hours with no success; one worker per AppFolio account.
- Sep 11: alerts once per customer, not per job; every check runs for every customer regardless of features bought; the first load is its own system; pull order is leasing/renewal/maintenance first, dashboard figures after; design for 10x today's customer count.
History windows (defaults per module)
| Module | Default window |
|---|---|
| Core entities (properties, units, tenants, leases, occupancies, availability) | Always complete |
| Leasing activity (guest cards, applications) | Open + 90 days closed |
| Maintenance (work orders) | Open + 90 days closed |
| Lease history | 12 months |
| Accounting — payments | Trailing 2 months |
| Accounting — financials | 12 closed months |
These are per-stage settings on the import record, configurable per customer.
Caveats
- A paginated result's "next page" link expires after 30 minutes; a bulk walk must finish inside that window or re-fetch from the start.
- A "slow down" (429) response often carries no Retry-After header; falls back to standard backoff.
- The Sep 11 bulk pilot measured a tighter real-world limit than AppFolio's documented rate, while the old per-property sync ran against the same account at roughly one request per second at the same time — re-measure once that interference is paused.
- The circuit breaker is process-wide, shared across every customer's checks in one run, not scoped per account; 429s were excluded from tripping it starting Sep 10.
File paths and job names
- Sync entry point:
lambda/appfolio-sync/handler.ts; schedules:lambda/appfolio-sync/schedules.json(16 jobs, one rate/cron each). - Today's per-property loop:
runSync(),handler.ts:2046, overloopPropsfromsyncConnections.buildTickOrder(...). - HTTP client / pagination:
src/lib/integrations/appfolio/client.ts—fetchPage/listAll/listAllWithMeta. - Resilience policy:
src/lib/platform/resilience.ts::appfolioPolicy— one shared retry/breaker instance for the whole process, not per account. - Today's onboarding "commit" step (property records only, no units/tenants/leases):
src/app/api/onboarding/pms/appfolio/import/route.ts. - Source files for this page:
design-outline.md,bulk-measurement.md,steady-state-audit.md,current-sync-map.md,appfolio-reports-api-facts.md,lookback-adrs-docs.md,lookback-prs.md,lookback-sessions.md(session research files, Sep 11 2026). - Onboarding import module (separate from the every-minute sync):
lambda/appfolio-sync/onboarding-import-runner.ts—MAX_IMPLEMENTED_STAGE = 5, self-invoking, gated byisOnboardingImportArmed. - Reset for a clean re-import:
scripts/reset-org-sync-for-reimport.ts(dry-run default,--apply --expect=<n>); guard override:scripts/lib/assert-test-property.ts::assertLiveCustomerOverride. - Portfolio-arm scripts (dry-run default, Fede-only
--apply):scripts/set-occupancies-portfolio-arm.ts,set-renewals-portfolio-arm.ts,set-guest-cards-portfolio-arm.ts,set-rental-applications-portfolio-arm.ts,set-lease-states-portfolio-arm.ts,set-balances-portfolio-arm.ts,set-work-orders-portfolio-arm.ts.
Job states — PARKED / DISABLED (schedules.json, Sep 12)
| Job | State | Reason (condensed) |
|---|---|---|
| work_orders | DISABLED | Parked to stop the 429 storm (~258 req/min vs AppFolio's ~28/min ceiling); a hand-disable didn't survive deploys until PR #7965 made state declared. Re-enable once the per-account budget is proven. |
| lease_states | DISABLED | Same storm, same fix path as work_orders. |
| renewals | DISABLED | Same storm; also the job this doc's engine chapter notes as PARKED, so its portfolio arm is inert either way. |
| vendors | DISABLED | Same storm, plus a second reason: saveVendor has no version guard yet, so an hourly sync could race-adopt ~822 unstamped vendor cards during the vendor-identity rebuild. |
| all others (10 jobs) | ENABLED | Running on their declared cadence class; six of them also carry a dark, off-by-default portfolio arm (Ch. 03c). |
A DISABLED entry must carry a stateReason — enforced by appfolio-sync-schedules.drift.test.ts — so parking a job is a recorded decision, never a silent one.
Build plan, PR by PR
Import chain
- 1. Import tracking record + progress read-out (data only) — small — proven by unit test, reading the record back.
- 2. Bulk report client: one call per report, walks result pages — small/medium — proven by bench test at Willows.
- 3. Stages 1–2 (portfolio, people & leases) behind a switch — medium — proven by bench test.
- 4. Stage 3 (leasing pipeline) + hand-off rule — medium — proven by bench test plus a read-only replay against Western Slope's real data.
- 5. Stages 4–5 (renewals & maintenance, dashboard numbers) — medium — proven by bench test.
- 6. Progress read-out on the setup wizard / settings screen — small — proven by screenshot check.
Everyday-sync chain
- 7. Switch each job to once-per-customer calls — one small PR per job family — proven by before/after request counts, bench test each.
- 8. Apply slower speeds, add the one shared "did this run" record, delete the old per-property path (only after Sep 18) — a few medium PRs — proven by before/after request counts at Willows.
Overnight PR list (Sep 11–12)
| # | What it does | Dark / arm name |
|---|---|---|
| 7899 | Pause switch | account pause flag |
| 7909 | Pause-aware sweeps and alarms | respects pause flag |
| 7907 | Import tracking record | data only |
| 7915 | Sync gate on import completion | hold until import done |
| 7908 | Portfolio walker plus in-process budget | in-process pacing |
| 7911 | Durable per-account budget | APPFOLIO_BUDGET#key |
| 7924 | Budget wired at the client choke point | APPFOLIO_BUDGET_ENFORCE#key, env kill switch |
| 7923 | Occupancies, whole portfolio | OCCUPANCIES_PORTFOLIO#key |
| 7933 | Renewals, whole portfolio | per-job arm |
| 7935 | Guest cards, whole portfolio | per-job arm |
| 7945 | Rental applications, whole portfolio | per-job arm |
| 7947 + 7949 | Lease states helper, then wiring | per-job arm |
| 7953 + 7958 | Balances helper, then wiring | per-job arm |
| 7948 | Import runner core plus stage 1 | stage-1 arm |
| 7952 | Sweep plus stale-stage reclaim | reclaim job |
| 7957 | Stage 2 writers | stage-2 arm |
| 7960 | Hotfix for a test escape | hotfix |
Not done, as of Sep 12 4 AM (superseded — see below)
- Import stages 3–5 (leasing pipeline, renewals & maintenance, dashboard figures) — done Sep 12–13, PRs #7967, #7978, #8057.
- Work orders converted to a portfolio-wide call — done, PR #8074 (window fix) + #8126 (portfolio arm, dark, unarmed).
- The wizard hook that starts the import on property confirmation — done, PR #8127.
- A customer-facing progress surface for the import — still not built.
- Schedules-as-data and a per-key lease — still the long-term evolution, not started.
- Western Slope's own bulk load measurement — in progress, Ch. 07b.
PR list additions, Sep 13 afternoon
| # | What it does | State |
|---|---|---|
| 8272 | Test-property fence — skip test properties on act_now/answer_live report jobs | Merged |
| 8275 | Re-enable the lease_states schedule for JP&Co | Merged |
| 8321 | Stage 5 financials fix — one 12-month call per building, plus a future-month request bug and an account-override read fix | Merged, 3:34 PM |
| 8322 | Stage 3 write concurrency — 8 properties at once, plus a heartbeat-write race fix from review | Merged, 3:59 PM |
| — | Unit-id fallback for the rent-roll mapper (the 75-unit gap) | In flight |
JP&Co staged arming — full sequence (Sep 12–13)
| Step | Time | 429s (jpco) | Verdict |
|---|---|---|---|
| Pre-arm baseline | before 18:27 | 75, then 55 / 2h | Storm-era baseline |
| + budget enforce armed | 18:27 | 37 | Clean drop |
| + occupancies armed | 21:24 | 10 | Clean drop |
| + guest_cards armed | 22:18 | 27 / 2.5h | Rose then re-settled; guest_cards itself showed 0 errors — rise was portfolio-wide, not that job |
| + balances armed | 08:55 | ~14/hr before → 7–9/hr | Balances job's own errors: 8/hr → 0 next hour |
| + applications armed | 09:46 | no alarm fired | Clean |
| + lease_states armed | 09:54 | no alarm fired | Clean; one 11-min warning self-cleared |
| + daily schedule re-enabled | 10:52 | — | Cutover complete; renewals stays PARKED by design |
Remaining 429s traced to test-property polling, removed the same day by the test-property fence. Renewals' own portfolio arm stays inert while the job is PARKED (schedules.json).
Source: prod logs /aws/lambda/propflow-appfolio-sync-prod (us-east-1), jpco filter; jpco-cutover-watch-2026-09-13.md.
Western Slope re-import — plan and result
- Approved (Fede, relayed Sep 12 evening, reaffirmed Sep 13: "Stop being so conservative and just finish it"): wipe Western Slope's old, incomplete AppFolio-synced data and re-run the new 5-stage import end to end.
- Step 1 — dry-run inventory: Run 2 measured 2,498 rows (up from Run 1's 2,359 forty-plus minutes earlier — coordinator ruled this is ~30h of incremental sync writes on the same org, safe to proceed on the fresh count).
- Step 2 — reset --apply: run as
--apply --expect=2498 --live-customer-org=<org>, ledgered override (PR #8064); 2,498 rows removed, preserved set (front door, denylist, pause row, credentials) verified intact after. - Step 3 — import, all 69 confirmed properties, 5 stages: Portfolio 1 min (234 rows → 159 units); People & leases 3 min (142 tenants/142 leases); Leasing pipeline 18 min (1,049 guest cards, 191 applications — 4s of AppFolio fetch, ~1,059s of our own per-property write loop); Renewals & maintenance 2.5 min (1,052 work orders, 380 vendors); Dashboard financials failed (~1,400 calls, 34% rate-limited).
- Step 4 — reconcile: 69/69 buildings; 159/234 units — 75 single-home doors dropped because the rent-roll mapper (
rowToRentRollUnit,occupancies.ts:397) ignores rows with a blank AppFolio "Unit" name and has no fallback to AppFolio's own numeric unit id, unlike the sibling unit-vacancy mapper which already does this. Root cause confirmed as real missing units, not placeholders (both sample properties checked have real doors AppFolio just labels differently). Fix: same fallback, ~10 lines + tests, targeted for tonight's re-run. - Result: sync pause lifted 2:25 PM on Fede's yes; everyday sync now runs on the portfolio engine with the per-account budget. First hour after: 429s ran 8/20/9/17/5 per 15 minutes with the pacer engaged — no storm, but several everyday jobs (tenant follow-ups, unit-vacancy checks) still poll per building rather than per account; under diagnosis, so team-facing before/after numbers are held until understood.
- Fixes shipped same day: financials stage cut to one 12-month call per building (~69 calls instead of ~1,400), plus a future-month request bug and an account-override read fix in the nightly financials sync (merged 3:34 PM); leasing-pipeline write concurrency raised to 8 properties at once, a projected ~8x on the slow part of that stage — review caught and fixed a real race in how progress gets recorded along the way (merged 3:59 PM). Unit-id fallback still in flight.