Do we need a build every time?

On 2026-10-11 14:42Z — about ten hours before this audit was asked for — Gera relayed Fede in propflow-docs commit 86778c9c6, verbatim: “We don’t want to slow down production shipping.” Batching release promotes is OFF and not to be proposed, with “every merge keeps shipping to production immediately”, and the move off the 8-core builder ruled out on the same grounds. I read that commit directly rather than taking it second-hand.

The batch window was nevertheless set back to 30 minutes at 2026-10-12 02:25:36Z, adding the exact cost Fede objected to: push-to-live median 5.0 → 22.1 minutes. The live configuration therefore contravenes a standing ruling, and the PR to make it the default is held at the merge gate under decision b91774562, raised by the session that owns those files. I did not raise a second block; that one is the live escalation.

What this audit adds points the same way Fede did. Remaining spend after yesterday’s fixes is 172 finished builds/day, $29.60/day, of which production promotes are about $10.95/day. The window change buys about $5.74/day of that. Fede’s note predicted it — an under-$15/day target “looks reachable without touching production cadence at all, which makes his constraint free” — and the measurements agree: the preview fix alone was $18.65/day, three times the larger lever, at no shipping cost. On this evidence the window does not need to move to hit the target.

Noted against myself: the first two versions of this page, and two replies in the #spending thread, presented the 30-minute window as a settled good without knowing of Fede’s ruling. Corrected here rather than quietly amended.

Cycle Sep 14 07:00Z → Oct 14 07:00Z — read from Vercel's billing API (/v1/invoices/upcoming), whose own periodStart/periodEnd carry those exact instants, i.e. Pacific midnight; the next cycle ends 08:00Z, so the boundary follows the zone and not a fixed offset: total $670.95, of which $562.55 is Build CPU Minutes on propflowai and $108.40 is everything else. Build CPU minutes price at $0.0035 each at our concurrent-builds tier (not in general — see below); the enhanced machine bills about 8 CPU-minutes per wall-minute. Every per-day dollar figure on this page is consumption, and the invoice charges consumption minus a $19.21/cycle allowance (~$0.64/day), so these figures sit about $0.64/day above what is actually billed — 4% at a $15/day threshold. The “$200 budget” is NOT one of these reads — it could not be found anywhere in the API and is still an inference from the wording of Vercel's notification email, so the “300% over budget” framing is unverified. Details below.

The window is a repo variable, PROMOTE_WINDOW_MINUTES, which overrides the file default in scripts/ci/promote-window.env. The switch was verified in the logs rather than inferred: promote run 38195214684 at 02:17Z reads window=5m; run 38196025454 at 02:30Z reads window=30m.

Why 5 minutes cost so much. A batch of work is approved and pushed to release 46–91 times a Pacific day, with a median gap of 13.6–18.2 minutes. A 5-minute window is below that gap, so it merged almost nothing — 0–1% of promotes were folded together, against 13–43% at 30 minutes. The rise was confirmed normalised, so the weekend cannot explain it: 0.10–0.12 → 0.27–0.32 builds per merged PR, and Sunday Oct 4 had 29 deploys against Sunday Oct 11's 80.

Replay of 328 real promote timestamps. Control: Oct 5–6 at a 30-minute window predicts 69 builds against 72 observed, 4.2% error.

The correction that matters: going back to 30 halves the rebuilds, but recovers only about $6/day — roughly 15% of the measured $37.07/day build-CPU line. The two larger items in the Oct 7–9 peak were both closed on 2026-10-11, four days after it. Nobody should expect the window change to undo the peak.

It was applied to 2,307 preview deployments over three days and let exactly 3 through, all correctly carrying [preview]. The dashboard really does run scripts/vercel-ignore.sh — read from the project API's commandForIgnoringBuildStep field, and confirmed in 22 build logs. The obvious field to read, ignoreCommand, returns undefined and would have produced the opposite answer.

Verdict: no. Cancelled builds did not bill, and the question is closed. Tested against the only authoritative billed figure that exists — /v1/invoices/upcoming, Build CPU Minutes 167,232 = $562.55 for the cycle.

And a refutation that needs no model at all: summing cpuTimeForBilling over only the rows returned for Oct 1–11 gives 223,990 CPU-min, which is 134% of the entire 28-day invoice. Eleven days cannot exceed twenty-eight.

The trap that caught two independent readings. cpuTimeForBilling is populated and non-zero on cancelled deployments — zero-count 0 of 2,236 sampled, minimum 8 CPU-min. Vercel computes it for every deployment and then leaves the cancelled ones off the invoice. A field named …ForBilling is not a statement that the row was billed.

The billing formula, verified on 3,057 real rows with 0 mismatches: timeForBilling = ceil(timeToContainerExit / 60s) × 60s, and cpuTimeForBilling = timeForBilling × cores. The timeToReady variant fails on 1,233 of 3,057 rows — so anyone summing ready − buildingAt is using the wrong field. Separately, /v2/usage (what the Usage page shows) reports wall seconds of timeToReady and includes cancelled builds. That is why the original thread figures of 1,065 and 449 matched a wall-second sum: real Vercel data, but not the billed quantity.

Independently confirmed by a second instrument, and the direction is the convincing part. A separate session's receipt (vercel-build-waste) matched the one Pacific day with an exact invoice line, 2026-10-10: excluding cancelled builds gives 7,512.0 CPU-min = $26.29 against an invoice of $26.29, +0.00%; including them overshoots by 180–200%. The day's arithmetic closes — 909 deployments minus 749 cancelled = 160 non-cancelled = 153 priced plus 7 that never got a container. Note which way that cuts: if cancelled builds DID bill, then excluding 749 of them would have UNDERSHOT the invoice. It matched to the cent instead. Two instruments, different methods, same answer.

What this does NOT settle, stated because the result is only as wide as its control. It settles the Build CPU Minutes line specifically, because that is the line $26.29 was matched against. If cancelled deployments were billed under some other line item, neither instrument could see it. That question is open. Also reconciled rather than overridden: the reading that concluded cancelled builds bill was not wrong about its facts — ~381 cancelled previews at ~0.3 wall-min each really did consume about 129 wall-minutes of container time. Vercel simply does not invoice consumed wall-time; it invoices cpuTimeForBilling, which a cancelled deployment does not carry.

The fix is live and working: cancelled deployments per hour went 49.1 → 35.5 → 1.3, a 96% drop across identical 11.23-hour windows. The ~32/day residual is branches cut before the fix, because Vercel reads vercel.json from the pushed commit's own tree. Self-healing; no action needed.

What still costs money: 172 finished builds/day = $29.60/day, split 37% production and 63% preview. After #18770 that leaves about $10.95/day of production promotes — at a measured release cadence of every 10–15 minutes.

The $200 budget could not be confirmed, and it is load-bearing. The only limit the API exposes is billing.controls = {analyticsSpendLimitInDollars: 500} — a Web Analytics cap, not a spend budget. Every Spend Management probe returned 404. So the "$200 budget" and the "300% over budget" framing rest on an inference from the wording of Vercel's notification email, and confirming it needs a person to open Settings → Billing → Spend Management. Until then it should not be quoted as a measured figure.

Two figures in the original Slack thread are wrong, against the billing reads in the summary above: the thread's cycle total of $609 is really $670.95, and its ~$48 for everything-but-builds is really $108.40 — an understatement of 2.3×. Its ~$560 for build minutes was right ($562.55).

Also found: /v1/usage is unusable (month-only, reproducible 500s), and Vercel exposes no per-day or per-project invoice line at all — so every per-day dollar figure in this audit, including the ones above, is computed from per-deployment CPU minutes against the cycle's price, never read from an invoice line.

We build on the 8-vCPU enhanced machine, so “move to a smaller one” is the first thing anyone asks after reading the numbers above. It does not work, and it fails three separate ways, any one of which is sufficient. Measured and closed by the vercel-build-spend lane; no PR was spent on it and the project is still on enhanced, read back.

1. The price per CPU-minute is flat across machines AT OUR TIER — and only at our tier. Vercel's buildCpuMinutes line has price.model: matrix over ownerConcurrentBuildsTier × machineType, read from /v1/invoices/upcoming: tier 1 is $0.0035 for standard, enhanced and turbo alike, but at tier 2 standard ($0.0030) is 14% under turbo ($0.0035), and at tier 3 ($0.0025 vs $0.00347) it is 28% under. Our effective rate is $582.18 / 167,352 = $0.003479, which places this team at tier 1 — the one row where the rate is flat. So a machine downgrade cannot move our price per CPU-minute and can only help by cutting billed core-minutes. It is not true that Vercel charges the same for every machine, and this ground stops holding if our concurrent-builds tier ever changes.
2. And it barely cuts those. Billing is ceil(timeToContainerExit / 60s) × 60s × cores. Fewer cores against a longer build partly cancel, and the whole-minute ceiling rounds against you on short builds — a build that took 70 seconds on 8 cores and 110 seconds on 4 bills 16 core-minutes either way.
3. The trial cannot be isolated from production anyway. buildMachineType is project-wide; the only per-deployment override is buildMachine: ["turbo"], a single-value enum. There is no way to put one preview on a smaller machine, so any experiment necessarily runs production on it.

Independently of all three, it falls under the same founder ruling as the window: a smaller machine means a longer build, and a longer build is a slower deploy.

• Oct 1–3 is gone. The project's deploymentExpiration.expirationDaysProduction = 7 has deleted every successful production deployment older than about 7 days while keeping cancelled ones for 30. Those days therefore read as 400+ deployments with zero successes, which is an artefact and not a fact. Positive controls confirmed the query works.
• Why the heavy lane's green runs quadrupled on 2026-10-07 — a co-cause of the build-count rise that nobody has explained.
• The configured spend limit under Settings → Billing → Spend Management, as distinct from the budget the billing API reports.
• Merge → live end-to-end latency, as opposed to release-push → live.

• Six documentation defects, with exact replacement text. The two worst: docs/runbooks/release-branch.md:62 says the heavy lane runs on a 30-minute cron when its cron is every 15 minutes and line 10 of the same file says 15; docs/runbooks/ci-and-deploy.md:158 says release is pushed "about once an hour" when the measured median gap is 14–18 minutes at 68–91/day.
• scripts/vercel-should-build.sh is dead code whose own header falsely claims to be the live dashboard hook. Its sibling scripts/vercel-ignore.sh is the opposite of dead and must stay: it is what cancels a hand-run vercel production deploy, which it did twice in the measured window.
• The #18772 fix applies per commit, because Vercel reads vercel.json from the pushed commit. Verified by pinning the crossover at promote 0ba405fb1; a long-lived branch cut before that commit still creates deployments.