Situs AppFolio budget: why it refuses, and why it was never pinned

What was wrong. The first revision of this page said the refusals were caused by an ~78-request hourly batch arriving “within seconds of each hour boundary,” and recommended spreading that batch as the cheapest fix. Both are gone from the page — deleted, not footnoted, because a caveat under a wrong causal claim leaves the wrong claim doing the work.

What replaced it, and by what measurement. A sibling lane on this dock measured dispatch directly rather than inferring it from report volume: 7,135 dispatch lines over 24h, folded onto minute-of-hour.

Nothing dispatches at :00. The hourly batch is already 21 minutes off the boundary, so whatever clusters at :00 — if anything does — the batch is not its cause.

Now open, and nobody should guess at it. Whether refusals cluster at :00–:01 at all is unresolved. The only reading is mine, and it is a 34-event sample over four hours (20 at :00, 2 at :01, 7 at :11, 3 at :16, 1 each at :36 and :41) that nobody has re-derived. It may be real with a cause other than the batch, or it may be small-sample noise. The prod-verify-24h lane is already reading the refusal population for its own baseline and will emit a minute-of-hour histogram as a by-product — that is where the answer comes from, and this page deliberately does not re-derive it, because two lanes computing one number is how one fact becomes two that disagree.

What this does NOT touch. Everything else here is measured per report, per row or per log line, and none of it depends on a clock story:

• The onset date is throttleCountSince — a counter’s birthday, not a condition’s start.
• 6,077 cumulative over 23.4 days; ~54/day now; both differenced reads.
• The budget is not pinned: 5.0 of 6 tokens, pausedUntil in the past, and nothing clears a pause because it expires.
• The 15-day paused by staff window and its 2,076-line-per-day positive control.
• Our own refusals at ~215/day with no counter anywhere, and the 82% unattributed-appfolio-client share.
• The 3.89× counter-to-log ratio and the noteThrottled branch that explains it.
• The per-report refusal-rate asymmetry — hourly-class reports 4–45%, once-a-minute jobs 0.8–1.8%. This is the measurement that still justifies the fairness fix, and parts 1, 2 and 4 stand on it unchanged.

It is real, and it is two separate numbers that the write-up merges into one. The budget row carries a counter for AppFolio refusing us; it carries no counter at all for our own budget refusing our own callers — and the second one is what starves work orders.

1a · AppFolio’s own 429s: 6,077, cumulative over 23.4 days — since-onset, not per-24h

This is the question the prerequisite PR’s reviewer flagged, and the answer is unambiguous: since-onset. throttleCount is documented in durable-account-budget.ts as cumulative and never reset, and throttleCountSince names the window it covers.

aws dynamodb get-item --table-name propflow-prod --consistent-read \
  --key '{"PK":{"S":"CONFIG"},"SK":{"S":"APPFOLIO_BUDGET#appfolio#situsgroup#86fe"}}'

So 6,077 spans 2026-09-29T16:00:52Z → now = 23.4 days, an average of 260/day. The current rate is nothing like that average. Two independent differenced reads:

The bulk of the 6,077 is therefore not a steady drip — it is front-loaded. CloudWatch confirms it, from the one log group that writes these lines:

aws logs start-query --log-group-name /aws/lambda/propflow-appfolio-sync-prod \
  --start-time <2026-09-28T00:00Z> --end-time <now> \
  --query-string 'fields @timestamp
    | filter @message like /situsgroup/ and @message like /noteThrottled/
    | stats count() as n by bin(24h) as day | sort day asc'

1b · Our own budget refusing our own callers: ~215/24h — no counter exists for this

This is the number that matters for starvation, and it is invisible on the status card. When acquire gives up it calls noteRefusal, which writes lastRefusedAt and exhaustedSince and no tally. The only durable record is a 15-minute-sticky flag — the exact mirror image of the pausedUntil trap the file already warns about, one over-reading and one under-reading. The refusal is visible only in a log line:

aws logs start-query --log-group-name /aws/lambda/propflow-appfolio-sync-prod \
  --start-time <now-24h> --end-time <now> \
  --query-string 'fields @timestamp, @message
    | filter @message like /account budget exhausted url=/ and @message like /situsgroup/
    | parse @message /caller=(?<caller>[^ "]+)/
    | stats count() as n by bin(24h) as day, caller'

Which pulls are losing, last 24h (215 refusals, parsed by report endpoint):

Refusals are spread across every report, not concentrated on one. Hourly over the same 24h they run 1–15/hour with a single 41/hour spike at 19:00Z — continuous all-day pressure, not one nightly batch. Work orders really are being refused (27 in 24h), which is the maintenance-intake starvation the dock was opened for; but the heaviest losses are the accounting reports (income statement, receivables, delinquency, rent roll, itemized rent roll: 133 of 215, 62%).

What I could not re-derive: the audit’s “126 other pulls skipped in the last 24 h.” My equivalent measurement on the same account and window is 215 budget refusals at the AppfolioClient.requestJson choke point. The 126 is presumably a count of sync skip rows (a coarser unit — one skip row can swallow several refused pages) from a different instrument. The two are the same family and the same direction; I did not find the query that produces exactly 126, so treat 215 as the receipt and 126 as unverified.

Read, not inferred. All three live accounts at 2026-10-23 01:21:50Z:

Every one of the three is nearly full and none is paused. version advancing 29 times in 6.6 minutes on situsgroup is independent proof that consumes are succeeding, not CAS-wedged.

The mechanics, concretely

The two wedges that could pin this budget — and the receipt that neither is in effect

Worth stating because a future reader will reach for them, and because the fix row must not break their defences:

1. A far-future pausedUntil. The ratchet only lengthens and nothing lowers it, so a malformed Retry-After written before MAX_PAUSE_MS existed would wedge every caller on the account permanently, unfixable without a hand write. Not the case here: pausedUntil is 5 minutes in the past, and the cap is in the code today.
2. A missing version attribute. tryConsumeOne reads version ?? 0 and then conditions on version = :expectedVersion, which fails forever on a row where the attribute is absent — five contention retries, then waitMs: 25, then a refusal, every call. The file’s own comment calls this out. It self-heals, because both recordThrottleEvent and noteRefusal seed version with if_not_exists. Not the case here: version = 67,453 and advancing.
3. A future lastRefillAt. elapsedMs = Math.max(now - lastRefillAt, 0), so a lastRefillAt written from a fast clock makes elapsed 0 forever; with stored tokens at 0 the bucket would read permanently empty and nothing heals it, since every other writer uses if_not_exists(lastRefillAt, …). This is the only shape that would actually produce “pinned empty.” Not the case here: lastRefillAt is 0.3 min in the past.

I widened git log origin/main over src/lib/integrations/appfolio and lambda/appfolio-sync across 2026-09-26…10-02. Nothing landed in either path between 2026-09-29T08:07Z and 2026-09-30T03:50Z — a window that brackets the onset hour by 8 hours before and 12 after. The two obvious suspects are both a day late, as the brief said: #14255 (budget always enforced) merged 2026-09-30T17:27Z, #14072 (arm at connect) 2026-09-30T14:09Z. “No code landed at that hour” is the finding, and the real cause is a config event.

The onset was the AppFolio connection for situsgroup being created, followed immediately by its first full-portfolio walk against an account that was in observe mode, not enforce. Four independent receipts:

So 2026-09-29T16:00:52Z is both things at once, which is exactly how the misreading happened: it is the first 429 this account ever recorded (hence the counter’s start) and a genuine onset of a real two-hour storm. What it is not is the start of a condition that persisted.

Then the sync was switched off for fifteen days

The gap in the daily 429 table is not missing data — the log group’s retention is 30 days, so 2026-09-23 onward is retained and 2026-09-29 itself returns 869 rows. Positive control, so a zero proves the sweep ran: querying 2026-10-05 for any line containing situsgroup returns 2,076 matched records, and every one of them is context PMSSyncLambda:

connection=org_a3376147-…#situsgroup not dispatched — paused by staff

one of 2,076 identical lines on 2026-10-05, once per minute; zero AppfolioClient lines that day

So the true timeline is: connect 15:58Z → storm 16:00:52Z–~18:00Z (869 429s, budget in observe mode) → staff pause 17:01Z, holding 15 days with zero AppFolio traffic → enforcement armed by hand 2026-09-30T04:23Z, then made unconditional in code the next day (#14255) → pause lifted 2026-10-14T20:05Z → continuous, genuine budget pressure from then to now: 20–55 real 429s/day and 165–602 self-refusals/day.

Six commits have touched this budget. Every one of them changed whether the budget is enforced, or how loud it is. Not one changed how the single bucket is shared between callers.

The caller-side PRs are the same pattern one layer out: #13008, #13084 (renewals skip on a spent budget), #14937 (rental_applications skips instead of paging Sentry), #15644/#15648/#15656 (renewals un-park). Every one made an individual caller fail quietly when the budget refused it. That is the right local behaviour and it is also how 215 refusals a day became a silent data gap instead of an alarm. A seventh PR that makes one more caller skip gracefully is the bandage this rule exists to refuse.

The measurement that should drive the design: the bucket is 81% idle and still refuses ~215 calls a day.

The surviving measurement: who gets refused, by report. Refusal RATE — refused over (served + refused), one 24h window. It is per report, derived from no clock story, and it is the strongest number on this page:

The jobs that pull once a minute are refused 0.8–1.8% of the time. The jobs the hourly class pulls in a batch are refused 4–45% of the time — a 5× to 50× difference across a 1,700× range of volume. A report pulled 1,431 times a day is almost never refused; one pulled 11 times a day is refused half the time. That is the fairness defect, and it is the whole case for the fix below. It also settles one thing in the negative: the three 1-minute act_now jobs are the budget’s biggest consumers (66.5% of volume) and are not what is being starved, nor can they be emptying the bucket — at one page per minute each they put 3–4 requests/min against a 24/min capacity.

Two conclusions do survive and do not depend on the trigger: raising capacity is the wrong lever (it moves the refusal to AppFolio’s side, where 6,077 429s say we are already at the ceiling), and slowing the walks is also wrong — utilisation is 19%, so there is no volume problem to solve.

Two cautions on this table. The “served” column counts GET log lines emitted after a token was granted (client.ts:1044), so a request that was retried inside its own ladder is counted more than once — which biases the computed rates down, not up. And tenant_tickler pulls at batch volume but is refused at act_now rates, so its placement is uncertain; it is shown where its refusal rate puts it, not where its volume does.

Four parts as first drafted. Part 3 is withdrawn (see the correction note); three stand, vertically sliceable, each useful alone, in the order I would ship them:

Two constraints the fix lane must not break. (a) computeAvailability is deliberately the only place the refill arithmetic lives — a per-class version must stay a single function with a third caller, not a second copy; its own docstring records that an earlier duplicate was wrong by a factor of capacity on the one card meant to be trusted mid-incident. (b) Every write in this module fails open and silent; a broken budget must never stop the sync. Both parts 2 and 4 add writes, and both must keep that.

Interaction with incremental-guest-card-window: that row attacks the demand side of one of the three top consumers — guest_cards at 1,431 requests/day, 21.8% of all Situs AppFolio traffic, re-walking the full window once a minute. It reduces the burst; it does not change who wins when a burst happens. The two rows are complementary and neither substitutes for the other.

• Whether refusals cluster at :00–:01 at all — and therefore the proximate trigger of the refusal-rate asymmetry. My only reading is a 34-event, four-hour sample, and it is superseded as a causal basis by the sibling lane’s dispatch histogram, which shows nothing dispatching at :00. The answer will come from the prod-verify-24h lane’s minute-of-hour histogram, emitted as a by-product of the refusal baseline it is already reading; this page deliberately does not re-derive it. See the correction note at the top.
• The audit’s “126 pulls skipped” figure — I could not find the query that produces it. My equivalent on the same account and window is 215 budget refusals at the client choke point. Same family, different unit.
• Where ~4,500 of the 6,077 counted 429s came from. This is the one number I can explain but not decompose, and it matters, so it gets the space. Logged AppfolioAccountBudget.noteThrottled lines since 2026-09-28 total 1,561 — 1,254 in the sync lambda, 307 in the Vercel runtime, 0 in the renewal worker (with a 1,935-line positive control), none elsewhere. throttleCount reads 6,077 over a slightly shorter window. The gap is ~4,500, i.e. 74% of the counter.
  
  The mechanism is in the code, not in a missing log group. noteThrottled’s pause write is lengthen-only, and on ConditionalCheckFailedException — a longer pause already standing — it calls recordThrottleEvent (which increments throttleCount) and returns without ever reaching the logInfo('pausing') line. So every 429 that arrives while a longer pause stands is counted and not logged, and that is precisely the case a storm produces — the module’s own comment says so, about the tally rather than about the log. Across the whole window the counter runs 3.89× the log. So throttleCount = 6,077 is the authoritative all-process figure and every log-derived 429 count on this page is a lower bound. What I did not do is prove the decomposition: I cannot show that the ~4,500 are all unlogged condition-losers rather than partly a process I did not find. A Fix lane should treat 6,077 as the number and the per-group split as indicative.
• Demand and refusal figures are sync-lambda only. The 6,574 requests/day, the per-report demand profile and the per-caller refusal split all come from /aws/lambda/propflow-appfolio-sync-prod. The Vercel app adds measurable traffic (307 logged 429s, ~30/day since Oct 15) that is not in those tables, so they are lower bounds. The minute-of-hour clustering and the 19% utilisation conclusion are unaffected in direction — adding traffic makes the collision worse, not better.
• Why 2026-10-19 spiked to 602 refusals (roughly 3× the surrounding days), all of it unattributed-appfolio-client. Part 1 of the fix would have answered this.
• Whether AppFolio counts API-key traffic and browser-session (L4) traffic against one pool — still unmeasured, as sections/appfolio-budget.ts has recorded since 2026-09-15 (asked as q-e03ad74b). If they share a pool, every number here is an undercount of the real pressure. This is a question for AppFolio, not a measurement we can take.
• Whether anything is still wrong with the Sep 29 storm’s downstream data. Out of scope for this row, but worth flagging: 15 days of paused by staff means Situs has a 15-day hole in work orders, balances and occupancy that no budget fix will backfill.

The four commands, in order

# 1 — the row, and all three accounts (read-only)
aws dynamodb query --table-name propflow-prod \
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" \
  --expression-attribute-values '{":pk":{"S":"CONFIG"},":sk":{"S":"APPFOLIO_BUDGET#"}}'

# 2 — 429 rate per day (widen --start-time to 2026-09-28 to see the storm and the gap)
aws logs start-query --log-group-name /aws/lambda/propflow-appfolio-sync-prod \
  --start-time $S --end-time $E --query-string 'fields @timestamp
    | filter @message like /situsgroup/ and @message like /noteThrottled/
    | stats count() as n by bin(24h) as day | sort day asc'

# 3 — our own refusals, by caller and by report
aws logs start-query --log-group-name /aws/lambda/propflow-appfolio-sync-prod \
  --start-time $S --end-time $E --query-string 'fields @timestamp, @message
    | filter @message like /account budget exhausted url=/ and @message like /situsgroup/
    | parse @message /reports\/(?<report>[a-z_]+)\.json/
    | stats count() as n by report | sort n desc'

# 4 — the config event that is the real onset
aws dynamodb get-item --table-name propflow-prod --key \
  '{"PK":{"S":"CONFIG"},"SK":{"S":"APPFOLIO_BUDGET_ENFORCE#appfolio#situsgroup#86fe"}}'
#   -> reason: "Situs connect 2026-09-29: 429 storm within minutes; ..."

Scope note, repeated because it is load-bearing: jpco, situsgroup and westernslopepm are separate AppFolio databases behind one login. Every number on this page is situsgroup unless the row says otherwise, and the jpco figures are present only as contrast. This page covers the Reports API path only — the L4 staff-portal browser path has no budget, no pacer and no counter, so nothing here is evidence about it either way.