0123 — Scheduled-job placement: one rule, and a reconciler for the Construct's manifest

• Status: Accepted (2026-08-26)
• Date: 2026-07-29
• Deciders: Gera
• Amends: ADR-0078 (the Construct — config/automations.toml is and stays the one manifest; this ADR adds a placement rule and a reconciliation loop to it)
• Supersedes: ADR-0003, ADR-0014 (its scheduling clause only — the stats-library extraction in 0014 stands)
• Related: ADR-0012 (the carve-out 0014 revoked), ADR-0001 (agent work doesn't run on Vercel), ADR-0116 (whose PO read-model depends on a schedule this check found missing)

Note on the Accepted flip (2026-08-26): the placement rule and reconciler described here are already the enforced standard in practice — the reconciler runs nightly via .github/workflows/schedule-drift.yml and annotates every PR touching the manifest/vercel.json — but it is a VISIBLE check, not yet a REQUIRED/blocking one. gh api .../branches/main/protection lists required contexts as exactly ["Build","Type Check","Unit Tests","review","Regression Gate — Leasing (required)"] (verified 2026-08-26); the "Schedule Drift" workflow is not among them, so a PR can currently merge on a red run. Flipping the status to Accepted records what the reconciler already does; it does not itself add branch-protection enforcement.

Context

PropFlow runs 105 declared automations across four runtimes — launchd 15, temporal 38, eventbridge 20, vercel 32 — plus scheduled GitHub Actions, which the manifest did not model at all when this ADR was written (closed 2026-08-26 — see the Known-gaps update below).

Counts re-measured 2026-08-02, on the manifest shipping in this PR, using parseManifestToml — the repo's own reader — rather than a grep. They read 99 / 19 / 27 when this ADR was drafted, and drifted across a 405-commit rebase. That is this document's own thesis landing on itself: the live_ref half of the same drift was caught, by the reconciler; the prose half had no mechanism and shows it. Numbers here are a snapshot with no reconciler behind them — treat the manifest as authoritative and re-measure before quoting. (A hand-rolled split('[[jobs]]') returns 107: it also matches the two [[jobs]] mentions inside this file's own comment header. Use the parser.)

That inventory already exists. ADR-0078 built it: config/automations.toml, "the single house every cron, daemon, detector, schedule, and Smith program is declared in," read by both runtimes. This ADR does not add a manifest and must not be read as proposing one — an earlier draft of this very change did propose a second one (config/scheduled-jobs.json), which was wrong twice over: it violated one-source-of-truth, and it repeated the exact failure it was written to catch, by not reading the inventory that was already there.

Two things were missing, and they are what this amendment adds:

1. A placement rule. The manifest records where each job runs. Nothing said where a new job should go, so the question got re-litigated per job and the ADR that did answer it was wrong (below).
2. Reconciliation. The manifest declares intent. Nothing compared it to reality, in either direction, so the two could diverge indefinitely and nothing would say so. Both incidents below are that shape.

The written standard and production disagree completely

ADR-0003 (Accepted, 2026-04-17): "Every scheduled job runs on AWS EventBridge Scheduler... Vercel Cron is not used. If the web app ever needs its own scheduled job, it also goes to EventBridge."

ADR-0012 carved out one exception for the KPI snapshot. ADR-0014 (Accepted) revoked it: "ADR-0003 applies in full again. Vercel Cron is not used." It even pre-wrote the rebuttal: "if someone proposes Vercel Cron again, point them at this ADR and ADR-0003."

Today: 30 Vercel crons run in production (vercel.json), against a standard permitting zero.

It is worse than a count. ADR-0003 §Alternatives explicitly rejected putting listings sync on Vercel Cron — "Rejected because listings sync is agent-side work (scrapes AppFolio, writes unit availability to DDB, consumed by Clara). Per ADR-0001, agent work doesn't run on Vercel." The single most specifically forbidden thing in that ADR is what ships: /api/cron/listings-sync runs every 15 minutes and propflow-listings-sync-prod-schedule sits DISABLED.

The rejected alternative won on the merits

That migration was not sloppiness. src/app/api/cron/listings-sync/route.ts documents it: cutover completed 2026-07-23, and "it must run here rather than the legacy EventBridge Lambda because the auto-unpublish hook needs APPFOLIO_BROWSER_AGENT_URL, which only the Vercel runtime has." Disabling the EventBridge rule was a planned cutover step, not neglect.

So an engineer hit a concrete constraint ADR-0003 had not anticipated, made the right call, wrote down why — in the route, where the next reader of that file finds it — and the ADR two directories away stayed wrong. There was no mechanism by which reality could update the standard, so the standard lost quietly, 30 times.

Nothing detected an absence, because nothing reconciled

The same missing loop hid a data bug. FinancialPeriod rows (the income statement) were written only by a human running scripts/import-income-statement.ts. For Camellia that ran once — 2026-03-25, from a canned 12-month PDF export — and never again. Every sibling AppFolio surface polls (balances 15m, work_orders 1m, lease_states 1m, occupancies 15m, vendors 1h); financials had no schedule at all.

Because the dashboard's trailing-N window slides forward with the clock while stored months stay put, the rendered series lost a month every calendar month. By 2026-07-28 "Trailing 12 months" was charting 8, and in March 2027 buildFinancialSummary would have returned undefined — a blank card whose label reverts to a confident "Trailing 12 mo". The data half is fixed (#4829: backfill of 98 months + AppFolio throttling, merged); the schedule half is proposed in #4835 (a daily financials job on appfolio-sync), still an unmerged draft at the time of writing.

Nothing alerted for four months, and the reason generalizes. The nightly metric-snapshot Lambda reads getFinancialPeriods and wrote fresh MetricSnapshot rows on time throughout, so the 26h cron-stale detector and the 04:30 freshness check (src/app/api/admin/dev/metric-snapshot-freshness/route.ts) were green the entire time. They watch the output layer. The staleness was one hop upstream, at the input layer, where nothing was watching.

A third data point on documentation's limits: while fixing that bug, an agent proposed putting the new job on Vercel Cron — which would have made it 26 — because it had not read ADR-0003. The ADR existed and did not prevent the violation it was written to prevent. That is the case for a reconciler rather than a stronger sentence.

Decision

One placement rule, and a reconciler over ADR-0078's existing manifest.

The rule

Placement follows what a job TOUCHES, not the fact that it is scheduled.

ADR-0003's rationale was right and its scope was wrong. It reasoned from ADR-0001 ("agent work doesn't run on Vercel") — sound — and then generalized from agent work to all scheduled work, which does not follow. Most of what accumulated on Vercel Cron is not agent work: it is in-process reconcilers and drift detectors over the app's own domain code.

The 30-second procedure

1. Does it need durable state across steps or time — timers, signals, retries? → Temporal. Stop.
2. Does it touch only the repo/CI? → GitHub Actions. Stop.
3. Does it need the mini itself? → launchd. Stop. (Rare, and a reason to be uneasy: it is the one surface with no cloud redundancy.)
4. Does it call an external system (PMS API, scrape, browser agent, third-party API)? → EventBridge + Lambda… unless it needs a runtime capability only Vercel has — an env var, an in-app secret, the email transport, in-process domain code. Then Vercel Cron, and write the reason in the route header and the manifest description. That exception is not a loophole; it is the listings-sync case, and writing the reason down is what keeps this ADR from going stale the way its predecessor did.
5. Otherwise — reads/writes only PropFlow's own data through app domain code → Vercel Cron.
6. Always: add a [[jobs]] block to config/automations.toml, with live_ref set to the live surface identity. The drift check fails otherwise, in both directions.

The execution-ceiling rule

No long-running external fetch inside a Vercel cron tick. A Vercel function has a hard maxDuration; a Lambda invocation and a Temporal activity do not have one at the same scale. When a tick can be killed mid-flight, any at-most-once stamp it writes before the slow call is a correctness hazard, not just a performance one.

This is not hypothetical. src/app/api/cron/lease-execution-notices/route.ts:24-32 records it: a single L4 executed-lease PDF fetch can run ~2 minutes when the runner has to drive an inline AppFolio MFA login first. On 2026-07-02 04:15Z the then-120s ceiling killed a run mid-fetch after the at-most-once stamp was written — "one new-lease notice: stamped, no email — the process died, so even the wrapper's degrade-to-no-attachment catch never ran." The ceiling is now 300s with a SWEEP_SOFT_BUDGET_MS that refuses to start candidates it might not finish, and per ADR-0098 the acquisition itself moved to a durable workflow.

The general form: durable workflows own acquisition; a cron tick may detect and enqueue. A cron that must do a slow external fetch inline is on the wrong surface, whatever else the rule says.

Ingestion jobs declare a max-staleness on their DATA

The financials bug was invisible because monitoring watched whether a job ran, not whether the data it produced was current — and the job that ran was downstream of the one that didn't.

Any job that ingests or derives data a UI reads must state the maximum age its output may reach before that is a defect, and something must alert on the DATA timestamp, not merely on job success. "The 04:00 job succeeded" said nothing about a source frozen since March.

The 30 existing Vercel crons, reviewed honestly

Grandfathered where they satisfy the rule; no migration project. Nineteen of them are unambiguous category 5 — reconcilers, drift detectors, digests and health reports over PropFlow's own data through app domain code. The rule places them where they already are.

Six touch an external system, so each needs a runtime-capability reason under step 4 or it is a violation. All six have one, verified in the route:

/api/cron/leasing-digest and /api/cron/leasing-report send email through the app's transport but read only PropFlow data — category 5 with an outbound side effect, not external ingestion.

Five more crons landed while this ADR sat unmerged, and this section is the grandfathering clause, so leaving them out would have shipped the rule with five jobs never placed under it. They are the same five the reconciler caught with no live_ref — which is why this gap is knowable at all. Placed now:

So of 30: 22 category 5, 8 touching an external system, each of the eight with a stated reason. One of those reasons (spend-snapshot) is weaker than the others and is labelled as such, because a grandfathering clause that grandfathers silently is the thing this ADR is against.

Two of the eight are the weakest fits, which is a different complaint from the weak reason above: spend-snapshot's placement rationale is sound-but- cost-based, whereas these two are a genuine tension in the rule itself. voice-call-reconciler and renewal-external-reconcile are genuinely polling a third-party API on a schedule, which is step 4's EventBridge case, and they stay on Vercel because what they write is deep in the domain layer. That is a real tension, recorded rather than argued away. Neither does a long fetch per tick, so the execution-ceiling rule does not bite; if either ever needs to, it moves.

The disabled EventBridge rule propflow-listings-sync-prod-schedule is a recorded migration, not an accident — but the manifest has not caught up. It is still declared enabled while the live rule is DISABLED, which is one of the four findings below. Deleting the entry would erase the evidence that a migration happened; the correct fix is enabled = false with the reason, which turns a surprise into documented history. (Its Lambda is slated for deletion per the route's own comment; that is separate cleanup.)

Enforcement

scripts/check-schedule-drift.ts reconciles live schedules against config/automations.toml in both directions, daily and on every PR touching a scheduling surface (.github/workflows/schedule-drift.yml). The comparison core (scripts/lib/schedule-drift.ts) is pure — given a live inventory and a declared one, produce a verdict — so it is unit-tested without AWS or the filesystem.

Five drift kinds, all fatal:

• unregistered — running, not declared (the "someone added a 26th cron" case).
• missing — declared, not running. The direction that matters most, and the one no "what is running" dashboard can structurally show, because the defect is the absence. This is the shape of the financials gap.
• schedule-mismatch — the cadence sets disagree. Compared set-wise, so a job carrying two cadences cannot silently lose one.
• disabled — live rule switched off while the manifest expects it running.
• unexpectedly-enabled — live rule running while the manifest records enabled = false. Without this, enabled = false is a blanket amnesty: re-enable the superseded listings-sync rule in the AWS console and it double-runs against its Vercel sibling forever, green.

Two mechanics worth stating because they are easy to get wrong:

• live_ref is explicit, never derived. The manifest id is a stable human key that deliberately does not match the live name — EventBridge rules add a -schedule suffix, Vercel ids drop the route prefix. A heuristic id→name derivation would mis-key entries and report phantom drift, so each reconciled entry carries live_ref. An entry on a reconciled runtime without one fails the check as unrunnable rather than silently dropping out of the comparison.
• Exit 0 in sync, 1 on drift, 2 when the check could not read its inputs. An unreachable AWS reports UNKNOWN, never "clean." A check that cannot fail is not evidence, and the failure class here is silence.

This landed red on purpose — and the red one has since been fixed in reality

On its first honest run (2026-07-29) the reconciler reported four findings. Three were defects in the DECLARATION and are fixed in this PR; the fourth was a real absence that must NOT be silently closed, so this PR was written to merge with the check failing at 1 finding. The redness is the finding; greening it by editing the manifest to match whatever happens to be live would rebuild the exact instrument that missed four months of a frozen income statement.

Update (2026-08-02, on rebase): the fourth finding closed the way a finding is supposed to close — the missing rule was created, not the declaration edited. propflow-appfolio-sync-prod-purchase-orders-schedule is now live at rate(15 minutes), ENABLED, matching its declared trigger, so the check now lands green at 0 findings across 50 reconciled jobs. Nothing in the manifest was relaxed to get there. Two caveats worth keeping honest:

• The empty-fetch prediction below (that the rule would throw every 15 minutes against the bench's zero PO rows) is untested here. The 24h Lambda Errors metric for propflow-appfolio-sync-prod is 0, but that Lambda serves all ten appfolio-sync rules and a caught-and-logged throw would not move it — that number answers a different question than the one the prediction asks.
• The ADR-0116 product question — should the property with actual purchase orders be opted in? — stands unanswered. A live schedule is not that decision; it just means the deploy ran.

The two rider findings are worth being precise about, because they are the reconciler finding a mistake in its own inputs. A job that rides another job's route has no schedule identity of its own; the fix is to point its live_ref at the host route, after which the set-union grouping collapses host + riders into one clean match (both riders declare their host's cadence). Root cause: live_ref was added to 52 entries and documented in none, so the convention did not exist to be followed. It is now in the manifest's schema header. That the check surfaced this at all is the mechanism working — a fabricated live_ref is exactly the kind of quiet inaccuracy a hand-maintained inventory accumulates.

Why purchase_orders stayed red (and what closing it actually required)

The naive read — "the rule is declared, so create it" — is wrong, and the check being blunt about what is missing rather than why is what forced the investigation. Verified 2026-07-29:

• The rule is genuinely absent; the nine sibling appfolio-sync rules exist. deploy.sh creates it unconditionally, so the deploy has not been run since #4733 added it — lambda/appfolio-sync has no auto-deploy workflow, unlike inbound-processor and listings-sync. That gap will recur for any future rule added to that script.
• The deployed Lambda does carry the PO code (confirmed by grepping the deployed bundle, with a negative control — LastModified alone proves nothing, since a config-only update moves it too).
• But the only property in vendorJobReferenceMode = 'purchase_order' is the test bench (appfolio-45, The Willows), which has ZERO purchase orders in AppFolio — the 84 real PO line-items belong to Camellia (75) and to property-less POs (9). The bench instead holds 9 seeded PO#demo-* rows.
• A zero-row fetch against a property with stored rows throws PurchaseOrderEmptyFetchError by design — the anti-mass-removal guard that stops one bad API response from flagging a property's whole PO book removed. So creating the rule today would fail every 15 minutes and log a discovery-budget breach every hour, permanently, about synthetic data.

So the schedule is not the real gap. The real question is a product one that belongs to ADR-0116 — should the property with actual purchase orders be opted in? — and it is not a drift fix. Closing the finding by deleting the declaration would have been the manifest-editing anti-pattern this ADR exists to prevent. What actually happened instead is the legitimate close: the deploy that had never run was run, and the rule now exists. The schedule-shaped half of the finding is therefore genuinely gone; the ADR-0116 opt-in question it exposed is not, and this reconciler was never the thing that could answer it.

This is the reconciler paying for itself. A dashboard would have shown nine green appfolio-sync rules and one absence nobody was looking for. The check named the absence, and the absence turned out to be a marker for an un-run deploy pipeline and an opt-in pointed at the wrong property — neither of which anything else in the system was watching.

Consequences

Easier

• A developer adding a scheduled job has a 30-second answer instead of five plausible options.
• A job that silently stops being scheduled is now detectable. It was not, at all, before — on any surface.
• Undeclared jobs fail at PR time, before they become archaeology.
• ADR-0078's manifest gains a feedback loop. It was already the catalogue; now it is a catalogue something checks.

Harder

• Adding a scheduled job now requires a manifest entry with a correct live_ref. That friction is deliberate and small; it is the entire enforcement mechanism.
• Two schedule expression dialects remain (EventBridge rate()/cron() with its 6-field quirk vs. 5-field POSIX). The manifest records each surface's literal expression rather than normalizing, so reading it needs no translation step — at the cost of not catching semantically-equal-but-differently-written pairs.
• The reconciler needs AWS read credentials, so the EventBridge surface is only checked on the nightly run, not on PRs.

The two costs this ADR knowingly accepts (ADR-0003's arguments, which survive and are not waved away)

• One pause mechanism. aws events disable-rule stops a job in seconds. Pausing a Vercel cron means editing vercel.json, passing CI, and deploying to production — the slowest lever we have, and slowest exactly when an incident makes that worst. Three mitigations, none of them complete: most Vercel crons are read-mostly reconcilers whose worst bad tick is a log line; the ones that act outward carry runtime kill levers that flip without a deploy (e.g. leasing-digest gates on LEASING_DIGEST_AUTONOMOUS_SENDING and a per-property flag); and a Temporal Schedule pauses in one API call. For a Vercel cron with no env arm, the honest answer is that this is a real regression against EventBridge, accepted with eyes open. A new Vercel cron with an outward side effect should ship with a runtime-flippable arm — that is what buys back the pause lever.
• One alerting story. ADR-0003's version was CloudWatch alarms on Lambda errors/timeouts, which exists only for Lambdas. ADR-0078 built the runtime-agnostic replacement — withAutomationRun records every run to AutomationRun regardless of runtime, and alerting = true fans a failure through the Operator — so the argument no longer distinguishes the surfaces the way it did in April. Coverage is close but not total, stated precisely: 27 of 28 /api/cron/* routes record a run (24 via withAutomationRun, 3 via defineDetector); /api/cron/participant-name-reconcile records nothing, so its runs are invisible to the Construct — re-measured 2026-08-02, and it is still the only one. And alerting = true is set on only 10 of 32 Vercel entries — most of the rest own a deduped sendBakeAlert of their own, but "most" is not "all" and nothing checks which.

Known gaps (stated, not hidden)

• The EventBridge read is narrower than "every schedule in the account." It lists rules on the default event bus in the ambient region only. A rule on a custom bus, in another region, or an EventBridge Scheduler schedule (a separate API — no in-repo use today) cannot be reported unregistered. That is the direction that fails silently: a missing finding is loud, but an undeclared job outside this scope simply never appears, which is the exact shape of absence this ADR is about. Latent at 20 rules; the fix when it matters is an explicit page walk plus a count assertion.

• Only 2 of the manifest's 4 runtimes are reconciled — eventbridge (20 entries) and vercel (32). temporal (38) and launchd (15) are declared but unverified: enumerating them needs a Temporal Cloud client and host access to the mini respectively. Temporal is the largest single runtime by count, so the failure class this ADR exists to close remains open on the biggest surface. UNRECONCILED_RUNTIMES names them in code and the check prints them on every run, so the claim "schedules are reconciled now" cannot be read as stronger than it is.

• GitHub Actions is not modelled by the manifest at all (AUTOMATION_RUNTIMES = launchd | temporal | eventbridge | vercel), so scheduled workflows are neither declared nor checked. Extending the manifest's runtime union is the natural next amendment and is additive.

  Update (2026-08-26): closed. AUTOMATION_RUNTIMES gained a 5th member, github-actions; scripts/check-schedule-drift.ts reads each workflow's own on.schedule[].cron straight from the checked-in .github/workflows/*.yml files (no network, same shape as the existing vercel.json read); and all 24 scheduled workflows are now [[jobs]] entries in config/automations.toml. npx tsx scripts/check-schedule-drift.ts --skip-aws runs green — 59 in sync (35 vercel + 24 github-actions), 0 drift. One gap this closure does NOT close: a workflow disabled via the GitHub UI/API rather than by editing the file reads identically to an enabled one, since nothing here calls the Actions API for workflow state — the same class of blind spot the EventBridge reader above already documents for rules outside its query scope.

• The check verifies a schedule exists, not that its job succeeds. CloudWatch alarms, AutomationRun status, and per-job detectors remain the answer for failures.

• lambda/appfolio-sync has no auto-deploy workflow. Its Lambda and its nine EventBridge rules only change when a human runs deploy.sh, so a rule added to that script stays unprovisioned indefinitely — which is exactly how purchase_orders went missing. This reconciler now detects that class after the fact; it does not prevent it. Giving that Lambda a path-filtered deploy workflow like inbound-processor's is the structural fix and is not in this ADR's scope.

• The manifest's existing entries were largely written from live state, so for the surfaces it now pins it asserts drift from here, not correctness of the starting point. The four findings above are the part of the starting point that was already wrong.

Alternatives considered

Re-assert EventBridge-only and migrate the 30 Vercel crons (ADR-0003 and 0014's position, held twice). Rejected on evidence. The argument for it is real — one pause lever, one alerting story, no execution ceiling — and the first of those is conceded above as a genuine cost. But ADR-0014's justification for reverting the carve-out was that the stats library had been extracted so a Lambda could import it. That worked for exactly one job. It does not generalize to 30 reconcilers that reach into the app's domain layer, data layer, email transport and secrets: each would need extraction into a Lambda-bundleable form or an HTTP self-call — the pattern ADR-0003 itself rejected — for no user-visible benefit. And it would move listings-sync somewhere it demonstrably cannot run. A standard reaffirmed twice and violated 30 times is not being under-enforced; it is describing a world that does not exist.

Write a second manifest for schedules (config/scheduled-jobs.json, the first draft of this change). Rejected, and worth recording as rejected: ADR-0078 already owns the inventory, a second one violates one-source-of-truth, and proposing it while writing an ADR about inventories decaying is the failure mode in miniature. What was missing was never the declaration. It was the reconciliation.

A dev-tools page listing all schedules. Rejected as the primary mechanism — it is read-only and requires someone to think to look, which is precisely what did not happen for four months. /admin/dev/agent-smith already renders the manifest (ADR-0078 P5) and did not surface any of this. It is a reasonable addition to the drift check, not a substitute. The reconciler is the thing that fails loudly on its own.

Normalize schedule expressions semantically so rate(60 minutes) and rate(1 hour) compare equal. Rejected: it needs a cron parser we do not have, and it would let the manifest drift from the literal string an operator sees in the AWS console — trading a real property (the manifest says what the surface says) for a cosmetic one.

Fail on any DISABLED rule, and treat enabled = false as an exemption. Rejected in both halves. A disabled rule the manifest documents as disabled is recorded history, not drift — but the inverse, a live rule the manifest thinks is off, is the more dangerous direction and gets its own kind (unexpectedly-enabled) rather than an amnesty.