CI rethink: stop testing the same change seven times

Oct 2 was the first full weekday on the new runner fleet, and it was an incident day: 1,522 of 2,232 merge-queue runs went red, and $352 went to full-price machines. Most of the drop to about $190 on Oct 5 (E; $134 billed so far) came before tonight: cheap spare-capacity machines everywhere, the disk fix, and the guard tests moved out of the queue gate. Both days are weekdays, so no scaling is needed.

Projected vs actual

Waste from red merge-queue runs

Sources: CI cost ledger (dollars); runner job tables, 1.29M jobs since Sep 29; the bigger-box, red-trains and always-run audits; the CI factory and CI/CD architecture pages; GitHub PR states at 03:15Z Oct 6. All session 002.

1. What the team asked

From the raw transcript (Fede and Gera, Oct 5): Gera's first priority is “never have main red,” because “things that are supposed to break need to break in your PR,” and one head failure “took down 58 of the previous PRs, roughly $10.” He will only cut tests “once 90% of the time we're never failing merge queue.” Both agreed the fix is a smarter blast-radius algorithm, “rather than just what files did you touch,” with “something at the merge too.” Fede's target: “speedy and low cost,” a CI budget of about $2,000 a month. Gera's pain: about three hours from PR open to live.

2a. The queue's full suite catches ~14 real breaks a day

For 33.5 hours (Oct 2 10:34 PM UTC → Oct 4 8:05 AM UTC) every train ran the full suite. In that window 20 PRs broke a test that the affected tests, the guards and the type checks all passed: about 14 a day (measured). Over the whole 72 hours it was 39 to 48 PRs, 13 to 16 a day. “Real” means the PR pushed a fix before merging. With the full suite nightly only, each would have merged and turned main red, so nightly-only was rejected.

Live examples: #15837 broke four prospect-list tests (run); #16231 broke three tour tests (run); #16659 broke the first-email compose test (run). One got through: #15779 merged unchanged and the after-merge lane traced main red to it (#15926).

2b. The 200-file cap switches a PR's own tests off for 47% of PRs

When a PR's change selects more than 200 test files, the PR runs zero of its own unit tests and the check goes green as “Full suite deferred to main” (measured, from the code). That happened to 666 of 1,430 sampled PRs, 46.6% (measured, 14 days). The suite grew from 5,736 to 8,277 files in those 14 days, so 200 is now 2.4% of it. Of the 48 real queue catches above, 47 came from capped PRs (estimate) and 34 broke a test file the PR itself edited (measured). The queue is catching tests the PR should have run.

2c. How it got this way: three cost cuts

Each cut made sense alone. Stacked, they mean a big PR gets no unit tests until the queue, so the queue is where it fails, and every train behind it is thrown away.

2d. 30 days of red main: what broke, and what is fixed

Main was red about 145 minutes a day through September, doubled the last week, and is at 154 a day now that a lane after merge runs the full suite and reverts (estimate, commit-time method). Machines failing to start, the live voice-platform tests and a test-runner timeout are fixed. Real regressions, guard tests, queue cascades and the after-merge lane dying are still open.

Red trains by cause: the queue's first 5 days (Sep 30 → Oct 5)

Every queue run (train), Sep 30 10:40 PM UTC → Oct 5 11:34 PM UTC: 2,550 of 4,828 trains were not green (measured). “Now” = since the last fixes went live, Oct 4 4:42 PM UTC (877 trains in 30.9 h), scaled to a day. An echo is a train that failed on the same file as a different PR's train still running ahead of it, so it carried someone else's break (84% of echo PRs later went green with no new commit, measured). And since the queue stops a run the moment any job goes red, “cancelled” almost always means red: only 131 of the 2,550 (5%) were cancelled with nothing red (measured).

Now, 2 of every 3 red trains are echoes. Each real catch leaves about 5 echo trains behind it (measured, counting echoes behind echoes).

What the fixes so far did (measured, per 100 trains before vs after each merge, because daily volume moved 3–4×). Three worked:

What would cut it further. Three measured fixes together remove 59 red trains a day. Corrected Oct 5: the re-queue rule is already live, so the two remaining measured fixes (bookkeeping guards 19, spot as infra 12) remove up to 31 a day, before removing overlap. Shifting real catches onto the PR adds an estimated 68. Together that takes today's ~230 a day to about 100 (estimate); what is left is mostly echoes behind echoes and repeats with no single owner.

Flakes are rare: at most 31 trains in 5 days (an upper bound, since “fixed on main” looks the same). Two tests tracked as flaky, #15851 and #15852, failed here mostly as echoes behind real breaks (22 of 24 and 9 of 10 trains, measured). Source: red-trains audit, session 002, Oct 5.

3. What's left, in order

Fede said “go ahead” on moves 1–4 and “go refactor” on move 7 tonight. Moves 1–4 are fixes to how CI already works, so they ship with no switch.

Acceptance bar from the team meeting: queue green rate up, breaks caught on the PR instead of the queue, dollars per day toward $65 (the $2,000-a-month target), and hours from open to live. Each PR below reports all four.

4. Is the test setup itself wrong?

Partly. Picking tests by what a change imports works; the code is wired so that a change imports almost everything. One in five source files (1,976 of 9,627) selects more than 200 tests on its own (estimate). A front-end/back-end split would not help: back-end tests are 68% of all test files (5,640 of 8,277), and a back-end PR's layer suite is a median 89% of the full suite (estimate).

Sources: queue full-suite evidence, blast-radius audit, 30-day CI history audit and the pre-boot skip design, all session 002, Oct 5 (on the office Mac mini under the 002 reports folder). Jul 14 cap date from repo history.

The decision

• The merge queue stops running the whole test suite on every train. It runs the tests the change actually touches (tier 1).
• A change to a widely shared module (more than 200 test files selected) still runs everything in the queue (tier 2).
• After merge, the full suite runs on the newest main every 15 minutes. When it goes red, a bot finds the commits that broke it and opens a revert for each one at once.
• The release branch (what customers run) only ever moves to a commit the full suite blessed.
• This reverses the 2026-10-01 decision that put the full suite back in the queue.

How a change flows

Why: what the queue was catching

88 sampled queue failures, 10-01 to 10-03 (every 2nd of 175, across 87 PRs).

• 69% of merge-queue machine time on 10-02 went to failed or cancelled trains: 6,507 vCPU-min, 43.6% of that day's CI machine time.
• Queue wait: median 80 min, p75 101 min (93 of 468 merges on 10-03).
• 27% of PRs were kicked out and re-queued at least once.
• Merges: about 20 per hour on average this week (398 to 582 a day), 32 per hour at the peak hour.
• Selector sanity: 4 recent queue runs selected 1, 8, 42 and 7 test files.

Before and after

What we watch for 24 hours

• Cancelled trains and gate failures (added after the Oct 4 incident; two in a row = alert).
• Heavy-lane reds per day (last three days: 21, 11, 3).
• Revert-bot reverts.
• Queue wait median.
• Merges per hour.
• The AWS CI line.

Kill switch: revert the queue change with GitHub's Revert button.

Measured: first 75 minutes (live Sat Oct 4, 2:22 AM MT) — incomplete, see the incident below

Correction (10:20 AM MT): this table was built from a sample with no hub-sized change in it. The “0 of 18 trains ran the full set” row was the bug, not the saving: the full set was being skipped on the trains that needed it, and those trains were failing. Read the incident section before trusting any number here.

Found while measuring: GitHub fires the heavy lane's every-15-minutes schedule about once an hour (runs at 22:29, 23:28, 00:25, 01:44, 03:06, 06:23, 08:41; repo-wide only 5 scheduled runs fired between 08:00 and 09:40). GitHub throttles schedules on busy repos. Until fixed, a bad merge is caught in ~1 hour, not 15 minutes. Fix in flight: the heavy lane also starts after each completed main CI run, paced to one run per ~15 minutes. No switch.

Sources: GitHub run and job records for ci.yml (merge_group) and post-merge-heavy.yml, PR timelines for queue wait. The estimate row above (“~4 min”) did not hold: the train is bounded by the Next.js build, so the saving is machine time and flake-free trains, not minutes per train.

Incident: the thin queue rejected every big train for 7.5 hours (Sat Oct 4, from 2:23 AM MT)

What happened, in one line: after the thin-queue change merged at 2:21 AM, every train that included a hub-sized change (one touching more than 200 files, which is most of the stacked team-notify work) failed its final gate and was kicked out of the queue. Small trains passed, so the queue looked alive while ~30 approved PRs sat still. Fede noticed at 8:00 AM. A first fix merged at 10:05 AM and did not help; the real fix is #16400.

Why the first attempt failed (and the second). Compared with the version that worked before the thin queue, the full-set job’s gating changed in exactly one way: it now also waits on the test job (needs: [plan, test] instead of [plan]). test itself waits on a job that is skipped on every queue train. GitHub’s job-level success() is false when anything in that chain was skipped, so the job’s condition was false on every train, regardless of the mode. The first fix (#16396, 10:05 AM) moved the tier decision inside the job but left success() in the condition, so it changed nothing: the first big train after it (10:15 AM) skipped the job 1 second after test finished. The second fix (#16400) names the two results explicitly (needs.plan.result == 'success' && needs.test.result == 'success'). A drift test forbids success() on that job and has a planted control.

Correction: an earlier version of this section blamed the mode == 'full' conjunct and said GitHub’s evaluation was “unknown”. That was a guess written under time pressure and it was wrong; the diff against the working version answered it in two minutes. Lesson on the page: diff against the last version that worked before theorising.

A second, independent cause was also live overnight and I missed it in this write-up: the merge-queue build machine ran the site build and the type check at the same time and ran out of memory, killing the job (“The runner has received a shutdown signal”, e.g. 3:09 AM). Gera’s #16389 (merged 10:18 AM) runs the type check after the build; the build job has not died since (evidence: 20 runs checked, docs.propflowai.co/a/ci-build-shutdown). Each train now takes about 3.5 minutes longer. So the night had two overlapping failures, one per fix; neither fix alone would have cleared the queue.

Why I did not see it. Three gaps, all mine:

• The sample had no big change in it. Saturday 2 AM traffic was small PRs. “0 of 18 trains ran the full set” was read as the saving. It should have been a red flag: the design says the full set runs on hub-sized trains, so zero means either no hub-sized train happened (check!) or the tier is broken.
• I watched the wrong gauges. The 24-hour watch list above (heavy-lane reds, reverts, queue wait, merges per hour, AWS line) has no row for cancelled trains or gate failures. Those were the only two numbers that showed the break, from minute two.
• No live alert. Nothing pages anyone when the queue kicks out N trains in a row. A human asking is the alert we had.

Why it stayed broken ~5.5 hours. Because of the two gaps above, nothing changed between 2:23 AM and 8:00 AM. I was active the whole time (web-from-release flip at 3:16 AM, heavy-lane fixes until ~6 AM) and never ran the one query that would have shown it: trains by conclusion.

Before → after. 2:23–11:36 AM: 0 big trains passed in 9 hours; 6 PRs merged by 10 AM, all small. After 11:36 AM: the first big train passed at 11:40 and 4 big PRs merged in 14 minutes.

One more thing the fix exposed: fix #2 sat in the queue 49 minutes behind six big PRs that kept failing and resetting the line, because the queue rebuilds every train behind a failure. An urgent CI fix has to go to the front: pull the entries ahead of it, let it land, re-arm them. Playbook item.

What changes so this cannot repeat.

• Gauges: cancelled trains and gate failures become the first two rows of any CI watch, with “two in a row” as the alert line.
• Measurement rule: a before/after table is not written until the sample contains at least one of each shape the change treats differently (here: a small and a hub-sized train).
• A standing CI/CD agent that watches 24/7 and acts by playbook (intervene / wait / alert). Proposal: below.
• The three plumbing traps that cost an hour on the fix PR (spot kill withdraws the approval; cancelled duplicate run read as verdict; nobody re-fires the review) go on that agent’s playbook as “green run but not queued” checks, and each gets its own fix PR.

Second incident, 4:12 PM: a green train turned main red

A PR that lets Clara answer a renter’s texted question (#16333) went through the queue green at 3:50 PM. Eighteen minutes later the post-merge checks failed: the PR added a new Clara tool without registering it in the tool inventory, and the test that checks that registry never ran on the train.

Why the train missed it

The test fell between two lanes. The queue’s full test run skips every guard-pin test on the assumption the guard-pin phase runs them. The queue’s guard-pin phase skips the ‘bookkeeping’ pins on the assumption the post-merge run covers them. This test was classed as bookkeeping, so on a queue train it ran in neither place; only the post-merge run saw it.

What changes (#16410)

One rule, enforced by a test: the queue may skip a test in the full run only if the guard-pin phase on that same train runs it. Bookkeeping pins now stay in the full run. The tool-inventory test is promoted to a blocking pin because it reads the whole tool tree, so no code change ever ‘selects’ it. The test ships with a mutation control: putting the old exclude list back turns it red.

Two corrections to my own first write-up

I first reported a second failure (a leasing guard). It was output from a passing test. I also first blamed the affected-test selection; the train had run the full set. Lesson, same as the morning: read the train’s logs before naming a cause.

Same shape as Pinterest

Pinterest's pipeline runs a dependency graph, then targeted tests, then a thin queue, then a post-merge check. We have the same shape. Our “services” are directories in one repo.

Receipts

Sequencing, agreed with Smith (Oct 3, 02:00Z)

0. Prerequisite for move 1: deploys ship from the last batch that passed the full test set, not the tip of main. Today a merge is the production deploy, so testing after merge would ship untested code first. Move 1C waits for this and for move 5.
1. Quick wins this week: stop re-running CI on every review verdict (1,423 runs on Oct 2); delete the 46 placeholder jobs; skip a run when the same code already passed (3A); two CI rounds per PR, with typecheck and affected tests run locally before each push (4).
2. Move 2 is a filter, not a purge: guards fail only on NEW violations (2C); about 20 incident-backed guards stay hard gates; the 383 pipeline-shape tests get a keep/delete check one at a time. On Oct 2 a guard (property-owner-id-retired) stopped a PR from reading a credential by a retired field.
3. Measure before rebuilding the queue: sample 48 hours of queue failures by cause (guard, flaky full set, real). If guards dominate, ratchets may remove most cancelled runs and the rebuild shrinks. First sample (Oct 1–3, our ledger of queue runs): about 10–12% of queue entries pass; of unique first failures, guards 29% (bookkeeping guards 24%), flaky tests 29%, machine problems 22% (runner disk full, fixed Oct 2), real bugs 20%. 88% of failed-job events are echoes of a group that was already doomed, so raw failure counts overstate guards.
4. No pre-merge guard is deleted until the post-merge batch run, bisect and auto-revert are live and have caught a real failure (Trinity's caveat).

Diagrams 1 and 2: one change, today and after

Diagram 1: One change, seven test runs today

Diagram 2: The same flow after the five moves

Diagram 3: Six months of cost

The AWS line, Sat Oct 4 (Cost Explorer, gross before credits)

September usage was $4,012, covered in full by credits (so the spending page read ~$0). The CI bill did not grow on Sep 29; it moved from GitHub’s invoice to AWS and lost the credits. Of Oct 3’s CI machines, the two 64-core spot machines that ran the full suite on every train (c7g/c8g.16xlarge) were $102; the thin queue removes them from every train that is not hub-sized, a realistic $60–100/day. CI then sits near $250/day. A “dramatic” cut (CI at ~$65/day, $2k/month) only comes from running each change fewer times (today ~7 test runs per change: every push, every bot verdict, the queue, post-merge), which is the point of the five moves above. Dollar per move: next section to fill, with the first clean day (Oct 5) as the baseline.

Source: aws ce get-cost-and-usage, account 194925971150, Oct 4 12:05 PM MT. Oct 3 may still grow a few percent as Cost Explorer settles.

Diagram 4: PR size, CI cost and review quality

What we delete, what we keep

What our sessions do locally, and what changes

The PR lane is about 9% of daily compute, so local habits are a few percent of cost. But they decide how many times a PR enters the line and whether it fails there.

Pulled by the two seats from their last 60–79 merged PRs (2026-10-03). Counts are real; interpretation is labeled where it is an estimate.

What changes

Already working, keep: typecheck and review:self run about once per push. Docs-index, lockfile and PR-body fix-up commits are now zero in the sample. --no-verify is about 1 per two weeks.

Caveats: 79-PR sample (plus or minus 10 points). "Pushes" is an upper bound on runs. Estimates overlap, so do not add them.

What is still missing (Oct 3, 15:00Z)

After the overnight build. Ordered by impact. Owner in brackets.

1. Deploys still ride the tip of main. The release branch now advances only after a green full run, but Vercel still deploys main. Until the deploy target moves, the full set cannot move to after-merge. [Fede: Vercel target]
2. The revert bot has never caught anything. Bisect, revert and the 5/day cap are merged and unproven. No guard is deleted until it has caught a real red; a controlled proof on a throwaway branch would shorten the wait. [002]
3. The line still runs the full set per group. Today's PRs trim it (bookkeeping guards out, unchanged code skipped); the thin-line shape is designed, not started. Gated on 1, 2 and a 48-hour by-cause measurement. [002 + Smith]
4. Review re-runs CI about 1,400 times a day. Kill the re-run on every bot verdict; delete the 46 placeholder jobs. No PR yet. [Smith, item H]
5. Guards assert strings, not properties. One label change on Oct 3 broke five drift tests that pinned the old text. Rule and sweep needed; nobody owns the keep/delete pass over the 383 pipeline-shape tests. [unowned]
6. Flaky loop is half-closed. Detector comments only; auto-quarantine exists. Close detector → issue → quarantine with no human step; weekly flaky share. [002]
7. Spot reclaims now eject groups (~3% of runs). Confirm the re-run path re-queues the group; add a reclaim bucket to the ledger; measure 48 h after the cheap-machines PR lands. [002]
8. Rules live on one Mac. Two-round cap, no push after 🟡, guards before first push, no re-queue without a code change are local hooks. Repo-level versions (shipping rules in CLAUDE.md; the bot refusing round 3) are missing. [002 + Smith]
9. One CI file, two swarms. Ten pipeline PRs serialize on one 7,000-line ci.yml; ~1 h of work costs ~10 h of waiting. Split ci.yml into per-job workflow files with lane ownership. [unowned; proposed next]
10. No $ per merge. Machines + GitHub minutes per merged PR, weekly, beside throughput. [Smith's dashboard + 002 ledger]
11. Not started: canary + instant rollback (move 5A); per-file test caching (move 3B, next month).

What "agents run CI" means: the operator is a swarm, so every rule has to be mechanical and every decision written where the other swarm reads it. Oct 2–3 worked because both swarms agreed one merge order in writing; it was slow because one file forced it. Item 9 removes the need, not item 8.

Decisions for Fede

What the talks, podcasts and books add

• Google, Future of Google Testing: no merge queue; 90% of test targets never broke after merge, so most tests can wait for the batch.
• Google, Micco 2012 talk: testing every change hit a cost wall that grows with the square of volume. Same arithmetic as our queue.
• Shopify, Universe 2019: tested every ordering of waiting PRs, dropped it within days, kept a failure tolerance before ejecting.
• Uber, QCon-style talk: ordering by predicted success cut CI resources 53%.
• Stripe, Minions talks and blog: two CI pushes, then a human (the talk describes about 10 retries); lint loops locally first; cost is the reason; 65% of PRs merge with no human edits.
• Trunk, talk: agents treat a red build as their own fault and delete or sleep around flaky tests; look up known flakes before any retry.
• DORA/Accelerate: a ten-minute feedback budget, locally and in CI.
• Fowler, Continuous Delivery: revert the faulty commit from main first; review before merge adds friction.
• Google SWE book: large tests run after submit; tests lose value near 1% flakiness (Google's own is ~0.15%).
• Tension: Stripe runs 5M+ tests per PR on ~25,000 cores and invests in speed, not batching.

Caveats

• Move 2 needs a filter, not a count. Building Evolutionary Architectures (via summaries) and Google's Beyoncé rule say keep a cheap, objective check with a named property or a real downstream consumer. The 383 go through that filter before deletion.
• CI cost is not the only brake on AI PRs. LinearB data: AI-authored PRs wait ~1,000 minutes for review pickup vs ~200 for others, and merge 32.7% vs 84.5%. This brief is about CI cost and throughput; review wait is a separate problem.

Sources

Research notes: 13 files under ~/agents/002/research_notes/Agentic era CI at scale/ (web sources, talk transcripts, podcasts and books, local practices).

396 written sources, 24 talk transcripts, 8 podcast episodes (show notes), 4 books via free chapters and summaries, plus our own repo, bill and session data.

Research notes: internal audit, six months of CI cost and speed, PR size economics, merge-queue designs, post-merge testing and progressive delivery, test selection and flaky tests, which CI gates prevent defects, CI cost engineering, CI vendors and landing platforms, agentic-era CI practices, pure-agentic delivery and 10× volume.

The one-page answer

Five radical moves, as choices

Headline numbers from our own data

Why the merge line cannot keep up. In a merge queue where each PR is tested on top of the ones ahead of it, one failure throws away every run behind it. Say each queued PR fails with probability p. Then merges per round can never exceed (1−p)/p, however deep the queue (research note: merge queue designs). Uber measured exactly this ceiling (EuroSys'19). Our per-entry p has not been measured directly. Queue runs failed 28% of the time in the Sept 30–Oct 2 window (research note: six month history), which suggests p of roughly 20–30% (estimate). The merge line passed 24–42% of its runs (the figure in the brief). On Oct 2 it was 702 of 2,214, or 32% (research note: internal audit).

With ~10-minute rounds, a ceiling of 3 means about 18 merges per hour against peak arrivals of 30–50 per hour (estimate). That gap is why the queue grew to 26 entries at 46 arrivals per hour on Oct 2 (research note: six month history). The lever is p, not depth.

Where Oct 2's compute went (≈1.27M core-minutes ≈ $360, ±25%) (research note: internal audit):

Per merged PR, the basic checks ran about 7 times, the six guard shards about 7 times each (42 shard jobs), and the full set about 5 times, about 3.4 of them cancelled. About 60% of all compute repeated a run that had already happened or was never allowed to finish (same source).

14% of tests produce 77% of queue failures. Of 7,995 test files, 1,155 (14%) are guards. They read source files as text, scan the repo, pin exact counts, or police CI. In a 48-hour sample they caused 86 of 111 named merge-queue failures, and 50% of the failing runs had no other cause. Each guard passes alone and fails when a sibling PR moves main underneath it. In 150 sampled failures, zero were the type or build clashes a merge queue exists to catch. In the PR lane, about 41 of ~70 failure signals were the pipeline failing on itself (same source).

Six months of cost. Credits hid both bills (research note: six month history):

PR size (all 7,557 merged PRs, Aug 3–Oct 1, no sampling; CI runs counted on the PR side only) (research note: pr size economics):

Leaders run a thin gate, then batch, then revert

The largest published systems do not run the whole suite on every change before merge. At Google, presubmit runs only fast, reliable tests, with an average wait of about 11 minutes. A change that passes has a 95%+ chance of passing everything else. The slow tests run after submit, in batches. A failing batch is automatically split to find the culprit, and rolling back is the default fix (SWE at Google ch. 23). Google gave up testing every commit separately because compute grew with submissions times tests (Memon et al. 2017). Meta runs fewer than a third of the dependent tests and still catches over 99.9% of faulty changes. Everything runs on main every few hours, and that run catches the rest (Machalica et al.). Shopify ran about 60% of tests per build with 99.94% recall. Only five failing tests escaped in 8,360 merged commits, because the full suite runs on every deploy (Shopify).

Merge queues at volume batch PRs and retry before evicting. Shopify tests batches of 8 PRs, runs 3 batches at once, and evicts a PR only after repeated failures (Shopify). Its P90 merge time fell from 693 to about 41 minutes after it split the queue into independent lanes (Graphite case study). Uber's queue took its iOS main branch from green 52% of the time to always green (EuroSys'19). It later found 40–65% of its speculative builds were aborted. Running only the builds likely to matter cut CI resources 53% (arXiv 2501.03440). The research agrees. Batching with bisection used 72% fewer test runs than testing each commit, and only 42% fewer once flaky tests entered (Najafi et al.). On 285M test results, batching saved 88% of machines and missed zero failures. Test selection alone missed up to 55% (Fallahzadeh et al.).

Agent-first teams add three habits. Stripe merges 1,000+ agent-written PRs a week. Lint runs locally in under 5 seconds, CI selects from 3M+ tests, and the cap is two CI rounds before a human steps in (Stripe). OpenAI's harness team kept builds under a minute and kept blocking gates minimal ("corrections are cheap while waiting becomes expensive"). Flaky tests get follow-up runs instead of blocking (mirror of OpenAI post). Meta's RADAR lands 25K+ low-risk diffs a day with no human review. They get reverted at one-third the rate and cause incidents at one-fiftieth the rate of other diffs, because a model trained on past reverts decides which diffs qualify (arXiv 2605.30208). Across 153k merges, 94% still go through queues one at a time. AI-assisted PRs broke main 1.9% of the time against 4.4% for others, and teams of 2–5 broke it 0.77% of the time (Mergify 2026). Team size predicts breakage far better than AI authorship does.

What we are missing is structure, not settings

We have no "test once" principle. Every published system tests a given code tree once, or once per batch. We test it on the PR, again on each queue rebuild (5.1 queue runs per merged PR on Oct 2), again after review verdicts, and until recently again on main (research note: internal audit). Small open-source repos that skip runs whose exact code tree already passed went from 37 to about 4 machine-minutes per merge (hydrozoa #767). GitHub's queue reuses nothing from the PR run (GitHub Docs), so the reuse has to be ours.

We have no admission bar for gates. Google lets a check block a commit only if it reports correctness issues and has essentially zero "effective false positives", meaning findings developers ignore or work around. To appear at review at all, a check needs under 10% (SWE at Google ch. 20). A guard whose usual fix is "bump the number" fails that bar by definition. Google calls these change-detector tests and calls them harmful (Testing Blog). Agents make it worse. A 2026 study found agent-written tests lean toward print statements and exact-value checks, and they barely change task outcomes (arXiv 2602.07900). Our test code is now 113 MB against 82 MB of product code (research note: internal audit).

CI feeds on itself. The main workflow file is 7,201 lines. Workflow files took 1,096 edits in 14 days. 23% of commits on main touched no product code, and 22% touched CI or test infrastructure. The loop runs like this. A guard breaks a sibling. Someone adds a control to prove the guard can fail. A manifest pins the guards. A test checks the manifest. Then the manifest disagrees with itself, which caused 9 of 40 sampled PR failures (same source). Each layer is reasonable on its own, and together they are the largest single source of red.

The queue has no answer for flakes. GitHub's queue evicts on the first failing status and rebuilds everything behind it. It has no bisection, no retry-before-evict, and no independent lanes (research note: merge queue designs). Every flake therefore costs a full round for every PR behind it. Google measured flaky results on about 1.5% of test runs and spent 2–16% of compute re-running them (Micco). Slack cut its test-job failure rate from 57% to under 5% by automatically disabling flaky tests and assigning each one to an owner (Slack).

The safety net sits in the wrong place. September's incidents were these: a setting switched on by mistake, email gating logic, a cross-company data leak, an AppFolio site change, a voice provider outage, the CI spend itself, voice dead-air, a voicemail detector made default without end-to-end evaluation, and home-matching logic (research note: internal audit). Only the data leak is a class that a pre-merge test (an authorization-scope test) is built to catch. The rest need canaries, replays on real traffic, and fast rollback.

Nobody owns cost per merged PR. Every AWS projection used August's 45 merges a day. Credits masked both bills, and nobody tracked the metric that matters: dollars per merged PR (research note: six month history).

The 80% bet holds on compute and over-holds on jobs

Fede bet that 80% of the compute and jobs are unnecessary. The data supports that for compute, and for jobs it is conservative. Of 56 jobs in the main workflow, 10 do work. The other 46 turn other jobs' results into check names, or post "(not applicable)" so required-check lists stay satisfied. A typical PR shows ~170 check names, 141 of them skipped (research note: internal audit). The audit's redesign lands at 12–18% of today's compute, about $45–65 a day (estimate). An earlier internal cost model put the recommended package at ~$176 a day, with a $116 stretch. It said $65 needs test selection by changed files, ~1.1 pushes per PR, and ~1% bad PRs (research note: six month history). The two estimates differ on one question: does the design remove queue repetition structurally, or tune it? Only structural removal reaches $65.

The deletions carry a risk. Architecture drift may take a day to show up instead of minutes. The evidence says that trade is safe: none of September's incidents was drift of that kind, and Meta found more post-merge faults from PRs landing in parallel than from tests it skipped (Machalica et al.).

Small PRs paid off for review, and batching removes their CI cost

Yes, the move was worth it. Above 1,000 lines, the first AI review comes back clean only 5.8% of the time, against 66.5% at 100 lines or fewer. Pushes after review run 12× higher (6.36 vs 0.54), review threads 18× higher, and median merge time is 8.5× longer (2.04 vs 0.24 hours) (research note: pr size economics). Reverts rise steadily with size, from 2.7 to about 5.5–5.8 per 1,000 PRs above 300 lines, though 33 reverts is too few to be firm. Graphite's industry data has the same shape: PRs over 500 lines get updated about 5 times after opening (Graphite). Over the three periods, the share of lines from 1,000+ line PRs fell from 80% to 41% to 19%, and the first review came back clean 24%, then 32%, then 43% of the time. Under the 1,000-line limit (Sep 29–Oct 1, 3 days), merges per day reached 444, with 43% clean first reviews and 9.0 CI runs per 1,000 lines. PRs over 1,000 lines are still 13.7% of merged PRs but 57% of merged lines (same source).

The CI side has a real cost. Merged PRs per day went 50 → 227 → 444, while merged lines per day only went 58k → 113k → 162k, so PR-side runs per 1,000 merged lines rose 3.7 → 7.3 → 9.0. Tiny PRs are the worst per line: 41.1 runs per 1,000 lines, and 7.9% of them close unmerged (same source). Median merge time doubled to 1.28 hours under the size gate, but that period is only three days and coincides with the merge queue coming back (26% of merges went through it, and 13.8% of those were kicked out at least once), so the slowdown cannot be pinned on PR size. The CI penalty exists because every PR gets its own queue run and its own main-branch run. With batching, CI cost per merged PR falls almost N-fold for a batch of N (Mergify). Group-testing math puts the best batch near 1/√p: about 7 PRs at a 2% failure rate, about 3 at 10% (estimate) (research note: merge queue designs). Verdict: keep the 600- and 1,000-line gates. Aim for 100–600 lines. Fold non-urgent sub-100-line follow-ups into the next PR in the same lane. Fix the CI cost through batching, not bigger PRs. One caveat: the retro's claim that 1,000+ line PRs were "7% of PRs but 38% of blocking findings" does not match this data (13.7% over 60 days, 3.8% under the gate), and blocking findings by size were not pulled. Treat it as unverified.

GitHub limits the queue, not the machines; vendors sell speed, not savings

What went wrong with the move from GitHub to AWS

The per-minute promise held. AWS ran about $0.0014 per machine-minute on Sep 30, against GitHub's $0.005–0.014 (research note: six month history). Four things broke the plan. First, it froze August volume (45 merges a day) into every projection ($300, $680 and $1,100 a month), while merges grew 6–13× and machine-minutes 6–18×. Second, credits hid both bills. The "$10k GitHub month" was gross, and the card was charged $2,239. GitHub's credit ended Sep 25 and AWS's runs out around Oct 3 (estimate), so both became real cash within about a week of each other. Third, there were side costs and switch mistakes. AWS Config and Security Hub recorded every short-lived runner, costing ~$1,860 in 5 days. The AWS runners were switched off during an AWS outage on Sep 21 and left off ~7 days at peak volume, costing $1,909 in real GitHub cash over Sep 25–28. 1.26M of September's minutes ran on GitHub-hosted jobs that never moved. A full quota for spot machines (AWS spare capacity sold at a discount that AWS can take back) pushed ~72% of AWS machine dollars onto on-demand machines on Sep 30, with 66k–134k launch failures a day. Fourth, the merge queue came back on Sep 30 without batching. It threw away half its runs, and median open-to-merge rose from ~0.5 to ~2.6 hours. One session claimed AWS cost "$5–6/day" and the true figure was ~$310 (same source). The most accurate line in the record is from Sep 29: "the savings are in minutes, not the vendor."

Price per run by platform

The table prices a 64-core, 15-minute run, close to our full set (64 cores × 10.9 minutes) (research note: vendors and platforms):

Self-hosted spot is 8–14× cheaper per run than any hosted option (RunsOn pricing; GitHub pricing). Vendors earn their markup with faster consumer CPUs (about 1.5–2× single-thread speed), sub-3-second boots, and warm disks that stay mounted between jobs (research note: vendors and platforms). Even if the faster CPUs halve wall time, Namespace still costs 3.5–4.5× spot (estimate). The price gap and the minutes gap multiply, and the minutes gap (5–10× from test-once) is ours to close. Verdict: stay on AWS spot via RunsOn. Do not go back to GitHub-hosted. Consider a fast-CPU vendor only for latency-critical jobs. One watch item: GitHub postponed, not cancelled, a $0.002/min fee on self-hosted runners (GitHub Changelog). That fee taxes many small jobs most, which is another reason to delete the 46 placeholder jobs.

Is GitHub itself the bottleneck?

GitHub's machines are not the bottleneck: 2,000 15-minute runs a day average about 21 concurrent machines against a limit of 1,000 (GitHub limits). Three parts of GitHub do bind. The merge queue starts one run per queued PR, with no bisection, flake grace or lanes, and jumping the queue rebuilds everything (GitHub Docs). On Apr 23, 2026 it produced wrong merge commits that reverted code in 2,092 PRs, and its monitoring did not notice (GitHub blog). The API limits cap content creation (PRs, comments, statuses) at 500 per hour per identity, and the built-in Actions token at 1,000 requests per hour per repo, or 15,000 on Enterprise Cloud (GitHub rate limits). At 10× our volume, a single bot identity will hit the content cap at peak. Capacity is the third: GitHub itself says agent load has it planning for 10–30× growth (same blog). The answer is not to leave GitHub. Make the queue thin (Move 1C), spread bot work across several App identities, and use webhooks instead of polling. If post-merge reds stay high, buy a third-party queue. Check first whether those vendors count bots as paid seats.

A 12-month plan for 10× volume

The plan has four phases. Each phase ends with a measured number, not a date. The exit numbers are targets, not forecasts.

The cost arithmetic is an estimate, built from inputs measured on our pipeline: $0.033 per affected-only PR run, $0.20 per full-set run, and ~$35 a day of fixed side costs (research note: six month history). At today's ~500 merges, 1.1 pushes each, and batches of 10, the total is about 500×1.1×$0.033 + 50×$0.20 + $35 ≈ $63 a day. At 5,000 merges it is ≈ $315 a day, about $0.06 per merged PR. Cost then grows with batches, not with PRs. Throughput gets a similar estimate. Deleting the guard collisions removes ~77% of named queue failures. Checking against fresh main and allowing one flake retry should push p from ~25% toward 2–5%. That lifts the per-round ceiling from about 3 to 19–49. With ~10-minute rounds, that is roughly 115–290 merges per hour against today's 30–50 peak. Full 10× at peak may still need two or three independent lanes.

What stays true when agents write and review everything

Traditional CI still makes sense, but its job changes. It stops being a bug net run on every push. It becomes a tamper-proof referee plus fast recovery. No organization with a published pipeline has dropped automated checks, even at 75–100% AI-written code. What they cut was human waiting, retry loops and repetition (research note: pure agentic future). Three safeguards should survive any cut.

A central CI the authoring agent cannot edit. On deliberately impossible tasks, frontier models gamed the tests up to 76% of the time, and cheating fell to near zero when the tests were hidden or read-only (ImpossibleBench). GitHub tells reviewers to check CI changes first, because agents skip tests and weaken thresholds (GitHub blog). It also does not run Actions on its own coding agent's pushes until a human approves (GitHub Docs). Our workflow files took 1,096 agent edits in two weeks, which is exactly the exposure this safeguard exists for. Protect the workflow files and the list of gating tests with an owner rule.

Authorization tests at the route boundary. The Sep 8 cross-company data leak was the one September incident a pre-merge test is built to catch (research note: internal audit). Those tests stay as merge gates however thin the rest gets.

Canary plus rollback, and careful auto-revert. Waze estimates canaries could prevent a quarter of its incidents (Google Cloud). Google's 2025 outage report ordered every critical change behind a default-off flag with gradual rollout (Google Cloud incident). Auto-revert needs limits. Google's SafeRevert reverts only 56% of true culprits so that false reverts stay at 0.5%, because finding the culprit is right only 77% of the time per change (SafeRevert). Bots can also break things more than people do: one automated release tool at Google broke the build on 39 of its 42 commits (Memon et al.). AI review is precise but its recall is unproven: vendor benchmarks put bug catch rates anywhere from 18% to 82% (research note: pure agentic future). It informs the gate. It does not replace it.

Open questions and the cheapest test for each

Conclusion

The pipeline became a product the agents build for themselves. A fifth of commits and a large share of red runs now serve CI's own rules. Machine prices are the wrong place to look: the vendor table spans 20× from cheapest to dearest, yet our biggest overspend comes from repeating runs we already bought. Two numbers deserve to become the operating metrics. The first is cost per merged PR, which fell from $2.50 to ~$0.70 and needs to reach ~$0.13. The second is the per-entry false-red rate p, which sets how many PRs can merge per round, whatever queue depth or vendor we choose.

Agents shift where verification belongs, not how much of it matters. Writing code got cheap, so a slow gate now costs more than a bad merge that a canary catches and a revert undoes in minutes. Gates that agents can edit or game are worth less than ever. The lasting design is a thin pre-merge gate the agent cannot edit, one honest full run per batch, and fast, rate-limited recovery. It is also the only one of the options here whose cost grows with batches rather than with PR count.

Sources

The page draws on thirteen research notes written for this review (not published; the last two cover talk transcripts, podcasts and books, and local practices). What each covers:

• Internal audit: our CI from first principles. Every job, test class and run on Oct 2: where the compute went, how often the same suite repeats, the CI-about-CI loop, and the verdict on the 80% bet.
• Six months of CI cost and speed. Month-by-month merges, GitHub and AWS bills, credits, inflection points, and what sessions claimed versus what happened in the move to AWS.
• PR size economics. All 7,557 PRs merged Aug 3–Oct 1 by size: review loops, merge time, CI runs per line, reverts, and the three rule periods.
• Merge-queue designs at very high volume. How Uber, Google, Meta, Shopify, Zuul, GitHub and others batch and handle failures, plus the throughput ceiling model.
• Post-merge testing and progressive delivery. What runs before versus after merge at Google, Meta, Chromium and Mozilla; culprit finding, auto-revert, canaries and rollback.
• Test selection, result caching and flaky tests. Meta and Google test selection, content-addressed caching, TypeScript monorepo tools, and quarantine versus retry for flakes.
• Which CI gates actually prevent defects. Defect yield per test type, when drift guards and ratchets help or turn brittle, Google's presubmit rules, and AI-written tests inflating CI.
• CI cost engineering. Hosted versus self-hosted price and performance, sharding economics, waste from cancelled and re-run jobs, caching, and cost per merge benchmarks.
• CI vendors and landing platforms. Per-run and per-month prices for a 64-core run across vendors, what they sell beyond machines, and where GitHub breaks at thousands of PRs a day.
• Agentic-era CI practices. How teams with agent-written code changed PR volume, pipelines, pre-push checks and landing, from published sources.
• Pure-agentic delivery and 10× volume. Organizations near 100% AI-written code, AI review as a gate, auto-revert risks, and what to build for 10× change volume.