01 · What happened (Sep 11)
Like asking someone who checks the mailbox once a minute to instead move an entire family's belongings into a new house, one box at a time, on foot.
- 25,000 requests to AppFolio in about three hours.
- 76% of that spent on every-minute jobs retrying "slow down" errors.
- Rent roll finished 46 of 69 properties; availability, 14 of 69.
- Still climbing at hour three, nowhere near done.
- Team paused the load by hand at 8:35 PM.
Proves: sync job logs, 5:15–8:35 PM, Sep 11.
Deeper → Appendix: the full measurement table
01b · Before and after, in API calls
Last night one truck made twenty-five thousand trips; this morning the same move rode in three.
- Old way: one call per property, per job, per timer — ~6,200/day for ONE property (math in appendix).
- JP&Co, 2 properties: ~12,400/day by that math — measured 13,910/day, close.
- Western Slope, 69 properties: ~429,000/day by that math — the 25,000-in-3-hours incident, throttled down.
- New way: one call per job for the WHOLE account — ~6,200/day, any portfolio size.
- Reduction: 2x for JP&Co, 69x for Western Slope.
Proves: schedules.json × property counts; Sep 11 logs, 13,910 req/day on jpco.
02 · Why: built for customers who already have data
A smoke detector is great at noticing a spark. It was never built to also be the fire hose for the first fire.
- Accreted job by job, one timer added at a time.
- The first pull was never designed as its own job.
- Sep 9 audit named a 3–5 customer ceiling in writing.
- That finding sat unactioned for two days.
- Completion records were retrofitted, not planned in.
Proves: written record — ADRs, PRs, session transcripts, Sep 9 audit.
Deeper → Appendix: the decision timeline
03 · The measurement that settles it
One truck for the whole move beats twenty-five thousand trips carrying one box each.
- A whole-portfolio call beats one call per property, always.
- Up to 5,000 rows come back per page, one request.
- About 7 requests per 15 seconds, per AppFolio login.
- Projected full first load: about 11 calls, not 25,000.
- Needs re-measuring once the old sync is fully paused.
Proves: bulk pilot, Sep 11 — rent roll and unit list, both clean.
Deeper → Appendix: the correction and caveats
03b · Architecture options, ranked
Recommended: option 1. Runner-up: 4, for the first load only, if stages ever branch. 3 is an additive later step, not a requirement.
1 · Lambda + DDB bucket
One Lambda, one bucket per account, gates the rate.
buys: no new services · costs: nothing
2 · EventBridge Scheduler
Per-account schedules replace the shared rule table.
buys: no-redeploy cadence · costs: unneeded console
3 · SQS FIFO
A queue decouples trigger from call.
buys: DLQ visibility · costs: new queues
4 · Step Functions (import)
Native per-stage retry, visible execution graph.
buys: native checkpoint · costs: 2nd orchestration model
5 · Fargate worker
Removes Lambda's 15-min ceiling; a fleet to run.
buys: no timeout · costs: new compute surface
| Option | New services | Isolation per key | First-load resume | Cadence change needs deploy? | Ops surface | Simplicity |
|---|---|---|---|---|---|---|
| 1 · EventBridge+Lambda+DDB | None | DDB bucket per account | Self-invoke + DDB checkpoint | Yes | One Lambda's logs + AutomationRun | 5/5 |
| 2 · EventBridge Scheduler | Scheduler groups/schedules | Same DDB bucket | Self-invoke + DDB checkpoint | No (API update) | + Scheduler console | 3/5 |
| 3 · SQS FIFO | Queue(s) + DLQ | DDB bucket (queue is not per-key) | Self-invoke + DDB checkpoint | Yes | + queue depth/DLQ dashboard | 3/5 |
| 4 · Step Functions | State machine | Same DDB bucket | Native state-machine checkpoint | Yes | + Step Functions console | 3/5 import, 2/5 refresh |
| 5 · Fargate worker | ECS/Fargate, ECR, IAM | In-memory, per task | Needs DDB checkpoint (memory lost on crash) | Yes, unless paired with Scheduler | + ECS task health | 2/5 |
Proves: architecture research, Sep 11 — steady-state audit: ~170k → under 18k requests/day.
03c · The engine, explained
Three speeds — right now, while you wait, end of day — one shared fuel gauge, and a switch for every gear that only Fede flips.
- Three speeds, defined once for everyone: act_now 1 min, answer_live 5 min, bookkeeping 1 hr.
- One shared budget per AppFolio account — every job spends from the same gauge.
- Each job's portfolio-wide call is its own dark switch, off by default, Fede-only to arm.
- A job can be PARKED (renewals, lease_states, work_orders, vendors) — survives deploys, needs a reason.
- New customers get a separate one-time import, residential properties only.
Proves: PRs #7966, #7911/#7924, #7923–#7953/#8126, #7965; residential ruling, Sep 12.
Deeper → Appendix: job states and file paths
04 · Onboarding, old vs new, in minutes
A moving crew that packs the kitchen before the garage and can pick up tomorrow, instead of one person carrying boxes at random until the building says slow down.
- Old way, Sep 11: 54/69 buildings failed, 146/234 units missing after 4h20m.
- New way: one-time 5-stage import, own budget, checkpointed, resumable.
- Willows bench, full chain: stage 1 7.6s, stage 2 11.0s (small test property).
- Stages 3–5 clean run: 2.8 minutes, 2,799 rows, 10 AppFolio calls.
- Full-portfolio result: see Ch. 07b — reconciled, with a known unit-count gap.
Proves: Sep 11 logs (54/69 failed, 146/234 units short); bench-first-load.md, run 2.
Deeper → Appendix: the five stages in detail
05 · What the customer sees
Like a package tracker: you see "out for delivery," not the truck's whole route.
- Starts as soon as the admin confirms which properties (default).
- Admin can close the laptop and walk away.
- Can sit dark for a few days before going live (Fede, Sep 9).
- One alarm per customer, only after a real failure.
- Never a flood of pages for one bad night.
Proves: Sep 9/10 rulings on alerts and stale-data handling.
Deeper → Appendix: the rulings list
06 · The everyday sync at 10x
One phone call to ask about every unit in a building beats calling once per unit.
- One call per customer per job, not per property.
- Seven jobs get a slower, still-workable cadence.
- One check removed outright; one folded into another.
- JP&Co already tops 1,000 rate-limit hits a day.
- 10x projection: ~170,000/day today's design vs ~14,000–18,000 proposed.
Proves: steady-state audit, Sep 11.
Deeper → Appendix: the per-job table
06b · JP&Co: fully on the portfolio engine
Turning every gas pump back on, one at a time, watching the gauge after each — until the whole station runs clean.
- JP&Co is fully on the portfolio engine: budget, occupancies, guest cards, balances, applications, lease states all armed; renewals stays parked; daily schedule went back on last.
- Balances errors: 8/hour before → zero next hour.
- Account-wide "slow down" replies: ~14/hour before → 7–9/hour after.
- No error or no-data alarm fired; one 11-minute warning cleared itself.
- Leftover "slow down" replies traced to test properties, fixed same day.
Proves: ops watch and prod logs, jpco account, Sep 13 (cutover-watch file).
07 · Decisions
Eight forks in the road: five settled, three still open.
| # | Decision | What's decided / default | Status |
|---|---|---|---|
| 1 | Build shape | Decided (Fede, Sep 11): EventBridge + sync Lambda, per-account budget in DynamoDB, portfolio-wide calls, first load as a checkpointed stage chain. Built. | Decided |
| 2 | Import start | Decided: on property confirmation. | Decided |
| 3 | Hand-off | Decided: after the whole import. | Decided |
| 4 | History depth | Decided: defaults per module, configurable. | Decided |
| 5 | Everyday cadences | Decided (Fede, Sep 12 AM): three global classes by report — act_now 1 min, answer_live 5 min, bookkeeping 1 hr. Declared in schedules.json. | Decided |
| 6 | Turn it on for JP&Co | Done, Sep 13: fully on the portfolio engine — budget, occupancies, guest cards, balances, applications, and lease states all armed, plus the daily schedule (Ch. 06b). Retiring the old per-job timers stays Fede's call. | Done |
| 7 | Reconcile with Gera's #7906 | Decided: #7906 closed; his declared-schedule-state idea shipped as PR #7965 (state survives deploys, mandatory reason). | Decided |
| 8 | Western Slope | In progress: Run 2 reconciled 69/69 buildings, 159/234 units (Ch. 07b); unit-mapping gap and financials-stage fixes in flight, sync pause lifted, everyday sync on the same engine. | In progress |
Row 6's last step, retiring the per-job timers, stays Fede's call.
Proves: Fede's decisions Sep 11–12; Sep 13 arming checkpoints.
Deeper → Appendix: history windows per module
07b · Western Slope re-import
Emptying a garage, then packing it again in order — and counting what's on the shelves before calling it done.
- Run 2 reset 2,498 rows, imported all 69 buildings, four stages clean: 1,049 guest cards, 191 applications.
- Reconciled 69/69 buildings, only 159/234 units — mapper drops a blank "Unit" name; fix in flight.
- Dashboard financials failed on throttling. Fixed 3:34 PM: one call per building, not ~1,400.
- Leasing write speed fixed 3:59 PM: 8 properties at once, ~8x faster.
- Pause lifted 2:25 PM; first hour clean — a few jobs still poll per building, under diagnosis.
Proves: reimport-result-2026-09-13.md (Run 2); unit-gap-diagnosis-2026-09-13.md.
08 · The road
Every new part of the plumbing gets tested with the water off before it's turned on for a real house.
- Six small PRs build the import, in strict order.
- Two more PRs bring the everyday sync to 10x.
- Every step ships dark, then gets tested at Willows.
- Then a read-only replay against real customer data.
- Nothing runs at a real property the same day it's built.
Proves: the standard rollout playbook — dark, tested, then Fede's call.
Deeper → Appendix: PR-by-PR proof plan
A1 · Appendix — full record
Every table, file path, and job name from the working record, for whoever builds this next.
Measurements by report (Sep 11)
| What | Result |
|---|---|
| Western Slope's real first load (one call per property) | 82 properties found, narrowed to 69 real ones; ~25,000 requests over 3 hours; 76% spent on 1-minute jobs retrying into "slow down" errors; only 2 of 16 jobs ever finished a pass |
| Bulk pilot — rent roll, whole portfolio, one call | 239 units returned, 5.2 seconds |
| Bulk pilot — unit list, whole portfolio, one call | 240 units returned, 1.5 seconds |
| Bulk pilot — next 3 calls attempted (tenants, vacancies, prospects) | all three rejected with "slow down" — likely the old sync competing for the same account's speed limit at the same time |
| Projected full first load, bulk design | ~11 total calls (one per report needed) — needs re-measuring clean, without the old sync running |
Before/after math, by the numbers
Old way = one AppFolio call per property, per job, on that job's own timer (schedules.json expression, unchanged by the cadence-class work). New way = one call per job, for the WHOLE account, regardless of property count (the portfolio arms in Ch. 03c/06b). Sum below is every Reports-API job in schedules.json at its declared cadence (excludes wo_enrichment/vendor_contact, which are L4 staff-portal sweeps, not Reports-API pulls).
| Job | Cadence | Calls/day, one property |
|---|---|---|
| work_orders | 1 min | 1,440 |
| lease_states | 1 min | 1,440 |
| renewals | 1 min | 1,440 |
| guest_card_messages | 2 min | 720 |
| occupancies | 5 min | 288 |
| rental_applications | 5 min | 288 |
| prospect_conversions | 5 min | 288 |
| balances | 15 min | 96 |
| guest_cards | 15 min | 96 |
| purchase_orders | 15 min | 96 |
| vendors | 1 hr | 24 |
| payments | 1/day | 1 |
| financials | 1/day | 1 |
| lease_states_history | 1/day | 1 |
| Total, per property, per day | 6,219 |
| Account | Properties | Old way (per-property × 6,219) | New way (portfolio, 1× 6,219) | Reduction |
|---|---|---|---|---|
| JP&Co (org_jpco) | 2 | ~12,438/day (theoretical) — measured 13,910/day, Sep 11 logs; 12% higher, self-stacking retries | ~6,219/day | ~2× |
| Western Slope | 69 | ~429,000/day (theoretical) — the 25,000-in-3-hours incident is that pace, throttled well below it | ~6,219/day | ~69× |
Source: lambda/appfolio-sync/schedules.json cadences, Sep 12 (renewals/lease_states/work_orders/vendors are PARKED as of Sep 12 but counted here at their declared rate — the "old way" this compares against is the design as built, not today's paused subset). Property counts: 2 JP&Co properties confirmed by a read-only prod audit, Sep 11 ~10:15 PM (supersedes an earlier, stale 25-property figure in the table below); 69 Western Slope properties from confirmed-property-ids.txt.
Steady-state (everyday) sync, today
| What | Result |
|---|---|
| JP&Co (25 properties, per the Sep 11 audit — later corrected to 2 real properties, see above) — rate-limit hits | ~1,000–1,200 real "slow down" responses per day, on a single, smaller account |
| Situs (94 properties) — rate-limit hits | close to zero — not proof it's safe at that size, just not loaded as hard yet |
| Occupancy check on a 69-property portfolio | ran 11 minutes, finished only 19 of 69 properties before time ran out |
| Wasted attempts per check, by job | occupancy 64% · tenant-balance 63% · renewals 45% · move-out 43% · applications 52% · prospect-card 54% |
Per-job recommendation and 10x projection
| Job | Today's speed | Calls/day today | Wasted | Recommendation |
|---|---|---|---|---|
| Maintenance tickets | every 1 min | ~3,200 | — | Slow to 3–5 min and once per customer |
| Move-out detection | every 1 min | ~19,500 | 43% | Keep 1 min, call once per customer |
| Renewals | every 1 min | ~19,500 | 45% | Slow to 5 min, once per customer |
| Occupancy | every 5 min | ~3,900 | 64% | Keep 5 min, fix first, once per customer |
| Rental applications | every 5 min | ~4,000 | 52% | Slow to 15 min, once per customer |
| Prospect cards | every 15 min | ~1,300 | 54% | Keep cadence, once per customer |
| Tenant balances | every 15 min | ~3,650 | 63% | Slow to 30 min, once per customer |
| Today (2 customers, ~119 props) | 10x, today's design (10 customers, ~940 props) | 10x, proposed design | |
|---|---|---|---|
| AppFolio requests/day | ~20,300 | ~170,000 | ~14,000–18,000 |
Timeline of decisions, with sources
| Date | Decision | Assumption it made |
|---|---|---|
| 04-16 | Read customer data via AppFolio's reporting tool; write changes through our own browser robot | Reads and writes on two separate systems, no shared bottleneck — never addressed a brand-new customer |
| 04-17 | Every scheduled check runs on one shared timer system (AWS EventBridge) | Checks are small and repeat forever, not a multi-step one-time job |
| ~07-xx | Each check's frequency reasoned about one job at a time | A customer already has data sitting there; a first load has nothing sitting there yet |
| 08-01 | Stop re-reading unchanged data on every check | Targets routine-checking cost, not a first-time load |
| 09-10 | Clara always answers with the last successful data; alert only after 6 hours; one worker per account | Root cause: a burst of "slow down" errors tripped a shared safety switch and silenced Clara for everyone |
| 09-11, evening | Every check runs for every customer regardless of features turned on | Triggered by the maintenance check failing 82 times for Western Slope, who doesn't use it |
| 09-11, ~8:40–8:55 PM | Fede, watching the load live: the first pull needs its own purpose-built system | "Our current design was 100% for when there's already data... checking every minute is completely wrong" for a new customer |
The one-line finding: no design document, PR, or planning session ever designed the first load as its own problem before tonight. The Sep 9 audit was the first to name the 3–5 customer ceiling in writing, and it sat unactioned for two days.
Rulings this design obeys
- Sep 9: every customer's checks fully separated from every other's; a login belongs to the company, never one person; onboarding is admin-confirmed properties, not automatic; AppFolio API access is turned on before setup begins.
- Sep 10: Clara always answers with the latest data however old; team alerted only after 6 hours with no success; one worker per AppFolio account.
- Sep 11: alerts once per customer, not per job; every check runs for every customer regardless of features bought; the first load is its own system; pull order is leasing/renewal/maintenance first, dashboard figures after; design for 10x today's customer count.
History windows (defaults per module)
| Module | Default window |
|---|---|
| Core entities (properties, units, tenants, leases, occupancies, availability) | Always complete |
| Leasing activity (guest cards, applications) | Open + 90 days closed |
| Maintenance (work orders) | Open + 90 days closed |
| Lease history | 12 months |
| Accounting — payments | Trailing 2 months |
| Accounting — financials | 12 closed months |
These are per-stage settings on the import record, configurable per customer.
Caveats
- A paginated result's "next page" link expires after 30 minutes; a bulk walk must finish inside that window or re-fetch from the start.
- A "slow down" (429) response often carries no Retry-After header; falls back to standard backoff.
- The Sep 11 bulk pilot measured a tighter real-world limit than AppFolio's documented rate, while the old per-property sync ran against the same account at roughly one request per second at the same time — re-measure once that interference is paused.
- The circuit breaker is process-wide, shared across every customer's checks in one run, not scoped per account; 429s were excluded from tripping it starting Sep 10.
File paths and job names
- Sync entry point:
lambda/appfolio-sync/handler.ts; schedules:lambda/appfolio-sync/schedules.json(16 jobs, one rate/cron each). - Today's per-property loop:
runSync(),handler.ts:2046, overloopPropsfromsyncConnections.buildTickOrder(...). - HTTP client / pagination:
src/lib/integrations/appfolio/client.ts—fetchPage/listAll/listAllWithMeta. - Resilience policy:
src/lib/platform/resilience.ts::appfolioPolicy— one shared retry/breaker instance for the whole process, not per account. - Today's onboarding "commit" step (property records only, no units/tenants/leases):
src/app/api/onboarding/pms/appfolio/import/route.ts. - Source files for this page:
design-outline.md,bulk-measurement.md,steady-state-audit.md,current-sync-map.md,appfolio-reports-api-facts.md,lookback-adrs-docs.md,lookback-prs.md,lookback-sessions.md(session research files, Sep 11 2026). - Onboarding import module (separate from the every-minute sync):
lambda/appfolio-sync/onboarding-import-runner.ts—MAX_IMPLEMENTED_STAGE = 5, self-invoking, gated byisOnboardingImportArmed. - Reset for a clean re-import:
scripts/reset-org-sync-for-reimport.ts(dry-run default,--apply --expect=<n>); guard override:scripts/lib/assert-test-property.ts::assertLiveCustomerOverride. - Portfolio-arm scripts (dry-run default, Fede-only
--apply):scripts/set-occupancies-portfolio-arm.ts,set-renewals-portfolio-arm.ts,set-guest-cards-portfolio-arm.ts,set-rental-applications-portfolio-arm.ts,set-lease-states-portfolio-arm.ts,set-balances-portfolio-arm.ts,set-work-orders-portfolio-arm.ts.
Job states — PARKED / DISABLED (schedules.json, Sep 12)
| Job | State | Reason (condensed) |
|---|---|---|
| work_orders | DISABLED | Parked to stop the 429 storm (~258 req/min vs AppFolio's ~28/min ceiling); a hand-disable didn't survive deploys until PR #7965 made state declared. Re-enable once the per-account budget is proven. |
| lease_states | DISABLED | Same storm, same fix path as work_orders. |
| renewals | DISABLED | Same storm; also the job this doc's engine chapter notes as PARKED, so its portfolio arm is inert either way. |
| vendors | DISABLED | Same storm, plus a second reason: saveVendor has no version guard yet, so an hourly sync could race-adopt ~822 unstamped vendor cards during the vendor-identity rebuild. |
| all others (10 jobs) | ENABLED | Running on their declared cadence class; six of them also carry a dark, off-by-default portfolio arm (Ch. 03c). |
A DISABLED entry must carry a stateReason — enforced by appfolio-sync-schedules.drift.test.ts — so parking a job is a recorded decision, never a silent one.
Build plan, PR by PR
Import chain
- 1. Import tracking record + progress read-out (data only) — small — proven by unit test, reading the record back.
- 2. Bulk report client: one call per report, walks result pages — small/medium — proven by bench test at Willows.
- 3. Stages 1–2 (portfolio, people & leases) behind a switch — medium — proven by bench test.
- 4. Stage 3 (leasing pipeline) + hand-off rule — medium — proven by bench test plus a read-only replay against Western Slope's real data.
- 5. Stages 4–5 (renewals & maintenance, dashboard numbers) — medium — proven by bench test.
- 6. Progress read-out on the setup wizard / settings screen — small — proven by screenshot check.
Everyday-sync chain
- 7. Switch each job to once-per-customer calls — one small PR per job family — proven by before/after request counts, bench test each.
- 8. Apply slower speeds, add the one shared "did this run" record, delete the old per-property path (only after Sep 18) — a few medium PRs — proven by before/after request counts at Willows.
Overnight PR list (Sep 11–12)
| # | What it does | Dark / arm name |
|---|---|---|
| 7899 | Pause switch | account pause flag |
| 7909 | Pause-aware sweeps and alarms | respects pause flag |
| 7907 | Import tracking record | data only |
| 7915 | Sync gate on import completion | hold until import done |
| 7908 | Portfolio walker plus in-process budget | in-process pacing |
| 7911 | Durable per-account budget | APPFOLIO_BUDGET#key |
| 7924 | Budget wired at the client choke point | APPFOLIO_BUDGET_ENFORCE#key, env kill switch |
| 7923 | Occupancies, whole portfolio | OCCUPANCIES_PORTFOLIO#key |
| 7933 | Renewals, whole portfolio | per-job arm |
| 7935 | Guest cards, whole portfolio | per-job arm |
| 7945 | Rental applications, whole portfolio | per-job arm |
| 7947 + 7949 | Lease states helper, then wiring | per-job arm |
| 7953 + 7958 | Balances helper, then wiring | per-job arm |
| 7948 | Import runner core plus stage 1 | stage-1 arm |
| 7952 | Sweep plus stale-stage reclaim | reclaim job |
| 7957 | Stage 2 writers | stage-2 arm |
| 7960 | Hotfix for a test escape | hotfix |
Not done, as of Sep 12 4 AM (superseded — see below)
- Import stages 3–5 (leasing pipeline, renewals & maintenance, dashboard figures) — done Sep 12–13, PRs #7967, #7978, #8057.
- Work orders converted to a portfolio-wide call — done, PR #8074 (window fix) + #8126 (portfolio arm, dark, unarmed).
- The wizard hook that starts the import on property confirmation — done, PR #8127.
- A customer-facing progress surface for the import — still not built.
- Schedules-as-data and a per-key lease — still the long-term evolution, not started.
- Western Slope's own bulk load measurement — in progress, Ch. 07b.
PR list additions, Sep 13 afternoon
| # | What it does | State |
|---|---|---|
| 8272 | Test-property fence — skip test properties on act_now/answer_live report jobs | Merged |
| 8275 | Re-enable the lease_states schedule for JP&Co | Merged |
| 8321 | Stage 5 financials fix — one 12-month call per building, plus a future-month request bug and an account-override read fix | Merged, 3:34 PM |
| 8322 | Stage 3 write concurrency — 8 properties at once, plus a heartbeat-write race fix from review | Merged, 3:59 PM |
| — | Unit-id fallback for the rent-roll mapper (the 75-unit gap) | In flight |
JP&Co staged arming — full sequence (Sep 12–13)
| Step | Time | 429s (jpco) | Verdict |
|---|---|---|---|
| Pre-arm baseline | before 18:27 | 75, then 55 / 2h | Storm-era baseline |
| + budget enforce armed | 18:27 | 37 | Clean drop |
| + occupancies armed | 21:24 | 10 | Clean drop |
| + guest_cards armed | 22:18 | 27 / 2.5h | Rose then re-settled; guest_cards itself showed 0 errors — rise was portfolio-wide, not that job |
| + balances armed | 08:55 | ~14/hr before → 7–9/hr | Balances job's own errors: 8/hr → 0 next hour |
| + applications armed | 09:46 | no alarm fired | Clean |
| + lease_states armed | 09:54 | no alarm fired | Clean; one 11-min warning self-cleared |
| + daily schedule re-enabled | 10:52 | — | Cutover complete; renewals stays PARKED by design |
Remaining 429s traced to test-property polling, removed the same day by the test-property fence. Renewals' own portfolio arm stays inert while the job is PARKED (schedules.json).
Source: prod logs /aws/lambda/propflow-appfolio-sync-prod (us-east-1), jpco filter; jpco-cutover-watch-2026-09-13.md.
Western Slope re-import — plan and result
- Approved (Fede, relayed Sep 12 evening, reaffirmed Sep 13: "Stop being so conservative and just finish it"): wipe Western Slope's old, incomplete AppFolio-synced data and re-run the new 5-stage import end to end.
- Step 1 — dry-run inventory: Run 2 measured 2,498 rows (up from Run 1's 2,359 forty-plus minutes earlier — coordinator ruled this is ~30h of incremental sync writes on the same org, safe to proceed on the fresh count).
- Step 2 — reset --apply: run as
--apply --expect=2498 --live-customer-org=<org>, ledgered override (PR #8064); 2,498 rows removed, preserved set (front door, denylist, pause row, credentials) verified intact after. - Step 3 — import, all 69 confirmed properties, 5 stages: Portfolio 1 min (234 rows → 159 units); People & leases 3 min (142 tenants/142 leases); Leasing pipeline 18 min (1,049 guest cards, 191 applications — 4s of AppFolio fetch, ~1,059s of our own per-property write loop); Renewals & maintenance 2.5 min (1,052 work orders, 380 vendors); Dashboard financials failed (~1,400 calls, 34% rate-limited).
- Step 4 — reconcile: 69/69 buildings; 159/234 units — 75 single-home doors dropped because the rent-roll mapper (
rowToRentRollUnit,occupancies.ts:397) ignores rows with a blank AppFolio "Unit" name and has no fallback to AppFolio's own numeric unit id, unlike the sibling unit-vacancy mapper which already does this. Root cause confirmed as real missing units, not placeholders (both sample properties checked have real doors AppFolio just labels differently). Fix: same fallback, ~10 lines + tests, targeted for tonight's re-run. - Result: sync pause lifted 2:25 PM on Fede's yes; everyday sync now runs on the portfolio engine with the per-account budget. First hour after: 429s ran 8/20/9/17/5 per 15 minutes with the pacer engaged — no storm, but several everyday jobs (tenant follow-ups, unit-vacancy checks) still poll per building rather than per account; under diagnosis, so team-facing before/after numbers are held until understood.
- Fixes shipped same day: financials stage cut to one 12-month call per building (~69 calls instead of ~1,400), plus a future-month request bug and an account-override read fix in the nightly financials sync (merged 3:34 PM); leasing-pipeline write concurrency raised to 8 properties at once, a projected ~8x on the slow part of that stage — review caught and fixed a real race in how progress gets recorded along the way (merged 3:59 PM). Unit-id fallback still in flight.