CI: the big picture

Why changes are slow and CI is expensive, and the big bets that fix it.

Nov 1, 2026 · for Gera and Fede · numbers are measured unless marked "estimate"

The machines aren't the problem; the process is. Our test machines are already about the cheapest available. The time goes to review rounds, and the money goes to checking the same code again and again.

What the numbers say

Slow changes are slow because of review rounds. Of the last 40 merged changes, those with 0–1 bot reviews took 31 min from open to merge. Those with 3 or more took 100 min, and that was about a third of them.
Cost comes from repetition. CI is 84% of the AWS bill. Each merged change runs CI about 5.6 times, and most of that re-checks code that hasn't changed.
CI has grown into its own product: 92 workflow files and about 76,000 lines of tests that only check CI.

The five big bets

Never check the same code twice. Adopt a build tool that knows what each change affects and caches results (Turborepo, using Vercel's built-in remote cache). This is the real payoff of splitting the apps. started, homegrown first step
Move safety from before merge to after deploy: automatic undo of a change that breaks main, plus a gradual rollout that watches error rates. Then we can test less up front. started
Get reviews right the first time. Run the review bot's same review on the agent's machine before pushing, and keep changes small. biggest open→live lever, not started
Deploy what CI already built instead of Vercel rebuilding it: about 5 min off merge→live (estimate). not started
Shrink CI itself: merge tiny jobs and delete low-value checks. started

Details: each bet, what we rejected, and the risks

1. Never check the same code twice

Today each step decides on its own what to test. A build tool such as Turborepo or Nx keeps a map of which code depends on which. It runs type-check, build and tests only for what a change touches, and stores results so a repeat run is instant. Turborepo's remote cache is part of Vercel, which we already pay for (check the current plan terms before relying on it).

It works best once the code is split into packages, which is exactly what the app split does. Step one is the homegrown per-test result cache already being built (estimate: $17–25/day and 5–10 min saved). Turborepo would extend the same idea to type-check and build.

2. Safety after deploy, not only before merge

Teams that ship fast don't try to catch everything before merge. They catch the common breaks cheaply before merge, then detect and undo bad changes quickly after. For us that means:

Automatic undo when main breaks (being built; today a person does it).
Gradual rollout: Vercel can send a small share of traffic to a new version and roll back if errors rise, using our existing error tracking.

Together these let us drop repeat test runs that cost $4–13/day (estimate) without more broken mains.

3. Fewer review rounds

Each round adds about 35 minutes (7 for the bot, plus about 27 to fix and push again, from Fede's data). Changes with 3 or more rounds take three times as long, and every push also re-runs CI. So this is both the biggest speed lever and a real cost lever. The cheapest fixes:

Mandatory local pre-review: run the same review prompt on the agent's machine before the first push. The tool already exists (/local-review); it isn't required yet.
Keep the 300-line habit so the bot reads the whole change instead of sampling it.

4. Deploy the build CI already made

After merge, Vercel builds the app again from scratch, about 6 minutes. Building once in CI and uploading the result (Vercel's "prebuilt" deploy) saves most of that. It needs one proof run, because environment variables must match.

5. Shrink CI itself

About 73% of CI jobs finish in under a minute but each starts a whole machine (about 37 s of overhead each). Merging them and moving 128 slots to cheaper ARM machines saves an estimated $500–750/month. More broadly: pause adding new checks, and delete checks that never catch anything.

What we checked and rejected

Idea	Why not
Kubernetes (EKS) for test machines	Net savings of about $500–900/month (estimate), but heavy upkeep. Machines aren't the bottleneck: one arrives in about 20 s.
Separate merge line per app (Trunk, Mergify, Aviator)	89% of changes touch the product, so the lines would barely split. It would mean rewiring about 90 places in CI plus our merge tools.
Hosted CI vendors (Blacksmith, Depot and others)	Estimated $2–8k/month more than our current machines (unverified prices).
Removing the public address on each test machine	The alternative (a NAT gateway) would bill our ~412 TB of monthly downloads, about $18k/month.
Dropping the merge line, or tests at arm	9 to 25 broken mains a day.

Risk to watch

GitHub has postponed, not cancelled, a per-minute fee for running CI on our own machines. At our volume that's about $1,700/month (estimate). Fewer, longer jobs reduce it.

Sources: CI baseline · Fede's CI plan · tracker · review-round timing from the last 40 merged PRs, Oct 31–Nov 1.