Every change is tested on top of everything ahead of it before reaching main, so main is always safe to pull; slow tests gate deploys, not merges.
2026-10-02 · for Gera · suite+undo live
Merges an hour
15→24+
Red main, min/day
~145→~0
Approved → on main
45 min→10–15
Assumes queue runs ≤10 min, under 5% kicked; see "The math".
This replaces a merge-on-green draft. Gera's objection stood: anyone pulling main before an undo got the bad change.
Today the PR runs the tests sitting next to the files you touched. Tonight a change to the renewal storage writers passed that way, and the renewal tests that use those writers never ran until the slow suite found them.
The fix is to pick tests by what depends on the change, not by where it sits. The code's import graph already knows which tests reach a file. Run all of those, plus the list checks, in 5–10 minutes.
The queue stays on and is the only way onto main. Each change is tested on top of everything ahead of it, so main only ever receives code that passed in that exact state. Pulling main is always safe. There is no separate "last green" to track: the tip of main is it.
What makes it fast: it runs the same 5–10 minute set the PR ran, not the full suite. What makes it cheap: a kicked PR cannot re-enter without a new push (5 PRs doing that caused 128 of 147 red queue runs on Oct 1), flaky tests stop blocking, and queue runs nobody needs are cancelled instead of hogging test machines. Once fewer than 3 in 100 PRs get kicked, 5 PRs share one run.
The full suite is slow and parts of it are flaky, so no merge waits for it. It runs once per deploy batch, on the same 30-minute rhythm production already deploys on. Green goes live.
Red means a change passed every test that depends on it and still broke something slower. The batch is halved to find that one change, it is undone on main, and its owner is woken with the failing test.
This is the one remaining gap, said plainly: such a change can sit on main for at most one batch window, about 30 minutes, and is never deployed. Testing by dependency makes it rare; the deploy gate makes it harmless to customers.
Status, 2026-10-02 16:00Z. Built and running as .github/workflows/post-merge-heavy.yml (propflowai #14606, #14790): every 30 minutes, one run at a time, the full unit suite and the full guard set on main's tip. Green moves the tag ci/heavy-green. Red re-runs only the failing files at the red commit (a pass there is a flake, nobody blamed) and at the last green (a fail there is environmental), then halves the merges in between to one culprit and opens [urgent] Revert "…" at once, armed for auto-merge, with a comment on the culprit PR; two culprits, a flake or a dead machine only open an issue. Since 2026-10-02 push to main no longer runs the full suite at all (#15194), and the merge queue runs the full set on one machine before a commit lands (#15399). Caught: the renewal-storage break from #14019 (issue #15293; that merge predates the queue's full set), auto-reverts #14906 and #15353 merged 63 and 40 minutes after opening. Since the queue's full set relanded (11:19Z): 8 lane ticks, 0 culprits, 1 flake — the measured marginal catch so far. Proposed, closed pending Gera's direction: #15523 (a revert landing on main starts the lane at once instead of at the next tick), #15519 (the 2-hour fallback revert titled [urgent] so the hold and the window recognise it). Not done, on purpose: deploys still key on each main push's fast checks plus the 30-minute window, not on ci/heavy-green; since the queue's full set already covers every commit, gating would add 30–50 minutes per deploy for the residual (live-vendor tests, known-red exclusions). While a culprit issue is open, pushes to main do hold their deploys.
Six of our checks keep a hand-written list in a file: routes, settings, scripts, a few counts. Nearly every PR has to add a line. When many PRs land close together, the file ends up wrong and the check fails in the queue or on main.
The fix is to stop keeping the file. The check builds the list from the code on the spot and compares the PR to main. Two PRs can both pass, and there is no shared line for them to fight over. Where a list of approved exceptions is genuinely needed, each exception gets its own tiny file.
Two doors go around the queue: an admin merge, and a reported GitHub bug where the plain API merge skipped the "require queue" rule. On Oct 1 one such merge broke main four minutes after the queue went off.
Whatever arrives on main without a queue run gets the same dependency-aware check on main; red means it is undone and its owner told. Every bypass merge is announced. Better still, remove the bypass: no bypass list on the ruleset, and rulesets rather than the old branch protection, which is where the API hole was.
| Gap | How often | What happens | Who notices |
|---|---|---|---|
| A change passes every dependent test and fails only the slow suite | Rare after dependency-based selection; tonight's #14019 is the shape. Counted from week 1. | On main for at most one batch window (~30 min), never deployed. Batch halved, change undone, owner woken with the failing test. | Deploy-batch run → the listener → the owner's session; posted in Slack. |
| The dependency graph misses a link (dynamic import, config, environment) | Unknown; this is why the slow suite still exists. | Same as the row above. | Same. Each case adds the missing edge to the graph. |
| A merge goes around the queue (admin, API) | 7% of red-main stretches over 90 days were "merged while red". | Same check on main; red → undone, owner told. Bypass announced; removed where possible. | Listener alarm on any merge with no queue run; Slack. |
| Someone pulls main just before an undo | Only in the two rows above, so a few times a week at most. | The listener tells every open PR branched after the undone change to rebase. | The PR's own session. |
| A flaky test in the queue | 3 of 30 own-fault queue reds in 24 h. | Retry once; passes-on-retry twice in 7 days → quarantined (still runs, cannot block); 14 days to fix or delete. | Test speed tracker; the CI factory page. |
| A test machine is lost mid-run | 9.4% of runs on cheap machines on Oct 1; 0 of 576 on full-price. | Queue and deploy-batch jobs run on full-price machines; a lost run re-runs, it does not kick the PR. | CI factory machine bays. |
| One bad PR re-enters again and again | 5 PRs caused 128 of 147 red queue runs on Oct 1. | No re-entry without a new push; hold label after two kicks. | Its owner session; the hold shows on the board. |
| Two PRs each fine alone, broken together | About 4% of queue entries on Oct 1. | Caught in the queue by construction; the later one is kicked, main never sees it. | The kicked PR's session. |
The queue is the only gate for merges. Main is not idle. Main runs the post-merge check on anything that arrived without a queue run (bypass merges), and the deploy batch runs the slow suite on main's tip every 30 minutes. Running the slow suite inside the queue is the July design, and it failed: every PR waited 12+ minutes and flaky storms kicked whole lines.
The same dependency-aware check on main, red → undo and tell the owner; an alarm from the listener on any merge that closed without a queue run; every bypass merge posted to Slack. And remove the bypass itself where possible: no bypass actors on ruleset 14925316, and rulesets rather than classic branch protection, which is where the REST-API hole was reported.
"Depends on the change" is the import graph, not the directory: tonight's #14019 changed storage writers and the renewal tests that import them lived elsewhere. Path-based "affected" selection skipped them; graph-based selection would not.
Docs-only and tests-only PRs skip the build and run only their own tests, but still go through the queue (it is the only door). Area lanes (Mergify "parallel mode", Aviator "affected targets") pay off only if a single queue saturates; at 20–24 PRs an hour against a ceiling near 50–100 an hour it will not. Revisit if the line grows.
Because every attempt failed on things this design removes, not on the idea: checks that ran only in the queue (now the PR and the queue run the same set), unfixed PRs re-entering (now blocked), cheap machines dying mid-run (now full-price for gate jobs), CI being rebuilt underneath it (now a change window), and the slow suite inside it (now at deploy). The retro's thresholds still apply: kick rate under 3–5%, runs under 10 minutes, machines lost under 1%.
Six checks caused this week's collisions: the route inventory, the guard-pin manifest, the record-only settings list, the eval row-count pins, the import-guard ratchet, and the unreachable-scripts baseline. They share one shape: a hand-kept file that nearly every PR edits, usually at the same line. Two green PRs each add an entry; main holds both and the file now disagrees with the tree, or the merge conflicts outright.
| Pattern | Use it for | How |
|---|---|---|
| Compute at check time | Inventories (routes, settings, scripts) and ratchets (import guard, pipefail, DDB escapes) | Derive the list from the tree during the check and compare to the merge base with origin/main, not to a committed snapshot. The check fails only when the PR introduces a new offender or removes a required property. No file to collide on. |
| Assert the policy, not the list | "Every route has an auth guard", "every setting has a writer" | The pin existed to force someone to notice a new entry. Replace it with the property it was protecting, checked for every entry. A new route that has a guard passes without anyone editing a list. |
| One entry per file | Allowlists that genuinely need human sign-off (a route that may stay unguarded) | guard/allow/<slug>.json, one entry each. Two PRs adding two files never conflict; git only fights over the same lines of the same file. |
| Regenerated by a bot after merge | True snapshots that must be committed (a generated schema, a golden image) | A post-merge job regenerates and commits with [skip ci]. Last resort: it adds commits to main every few minutes and still races. Prefer the first two. |
| Counts and totals | Row-count pins, "N escapes allowed" | Never. A stored total is a sum two PRs both change. Replace with a set of named offenders keyed by path (first row), or a fixed threshold with no per-file exceptions. |
Cost of this change: it is six tests, each a few hundred lines, and it is the single largest removable cause of red main after "never ran on the PR" (19 red stretches a month in the 90-day count). The guard-kinds file was already made non-fatal in a batch (#14855); this finishes the class.
| Stage | What changes | Speed | Red main |
|---|---|---|---|
| Week 1 make the queue fast |
Test selection by import graph on the PR and in the queue (replaces path-based "affected"). The six list checks become compute-at-check-time. No re-entry without a push; hold after two kicks (label + listener). Orphaned queue runs cancelled (concurrency keyed on the queue ref). Queue jobs on full-price machines. Flaky quarantine from retry evidence. CI changes only in a set window, each trialled. | Kick rate 14% → under 5%; queue run 12 → ≤10 min; approved → on main 45 → 10–15 min; throughput ceiling ~24 → ~50/h (see the math). | Main stays green by construction. Slow-suite-only misses still surface on main until stage 2 moves the suite to the deploy step. |
| Weeks 2–3 move the slow suite |
Full suite moves from every push to main to once per deploy batch (≤16 merges or 30 min), with halving to a culprit, undo, and wake-the-owner. Post-merge check + undo for bypass merges; bypass removed from the ruleset where possible. Docs-only and tests-only PRs skip the build. | No merge ever waits on the slow suite. Full-suite machine time stops competing with PR runs (~750 min/day today). | A slow-suite-only failure is on main ≤30 min and never deployed; every red names a culprit. |
| Later | Batch 5 PRs per queue run once the kick rate is under 3% (one run per 5 merges instead of 5). Area lanes only if the queue saturates. Success-probability ordering (Uber's model) only if batches grow past 5. | Queue cost per merge ÷ ~4. | Unchanged. |
Not in this plan: a stack-aware queue (part 2 branched off part 1). It conflicts with squash merging, and a PR based on another branch gets no CI run here. A 10–15 minute merge is the agent-native answer.
If a share p of PRs fail on their own, a batch of 5 passes with probability (1−p)5:
| Per-PR kick rate p | 5-PR batch passes | Where it comes from |
|---|---|---|
| 14% | 47% | Oct 1, measured. 5 PRs re-entering unfixed, plus checks that only ran in the queue. |
| 8% | 66% | Own-fault kicks only (30 of about 380 entries over 24 h). |
| 5% | 77% | Mergify's "high failure rate" line: batches of 2–3 at most. |
| 3% | 86% | The retro's threshold. Reached by moving the 21 queue-only catches to lane 1 and blocking unfixed re-entry. |
| 2% | 90% | Mergify's "low failure rate" line: batches of 5–10 are fine. |
| p | 12-min runs (Oct 1), c=10 | 5-min fast set, c=10 | Versus 20–24/h arriving |
|---|---|---|---|
| 14% | 24/h | 57/h | Today's rate: the line grows at 12 min, barely clears at 5. |
| 5% | 38/h | 91/h | Thin in bursts at 12 min; 4× headroom at 5. |
| 3% | 42/h | 100/h | Why "fast set only" matters as much as the kick rate. |
Ceilings, not forecasts: they ignore lost machines, line-jumps, and rework, all of which happened on Oct 1. Model from the retro; the "CI Checks" job measured 9.0 min in a cold queue vs 3.4 on a PR, so the 5-minute cycle assumes warm caches (#14950).
This is the arithmetic behind the earlier draft, kept because it is why the queue stays: without it every collision is 5–10 minutes of red main that anyone can pull. How many collisions a day there really are is still open. The 30-day main sample says 15% of 264 episodes, about 1.3/day. The Oct 1 queue caught 19 collision-or-stale-green runs in 455, about 4% of entries, which at 450 merges a day would be closer to 15/day if every one surfaced on main. The truth is between, and week 1's listener count settles it:
| Collisions/day | Red minutes/day (lane 3) | Share of a 16-h day | Call |
|---|---|---|---|
| 1–2 | 10–20 | 1–2% | Small, but each one is pullable. |
| 3–4 | 30–40 | 3–4% | With the queue: 0 by construction; a merge waits one run. |
| 10+ | 100+ | 10%+ | Same as today; the queue is mandatory and area lanes become worth it. |
The first draft of this page (2026-10-02, morning) recommended turning the queue off, merging on green, and undoing a bad merge within 10 minutes once main's own checks caught it. Gera's objection: anyone who pulls main in those minutes gets the bad change, and a "last green main" pointer does not help if green means only the fast checks, because the slow suite can still find something later. And if you have to wait for full green, merging fast is pointless.
All three points hold. The revised design answers them in order:
What the first draft got right is kept: the list checks that cannot collide, flaky quarantine, no re-entry without a push, full-price machines for gate jobs, cancelling orphaned runs, and the undo-and-wake mechanism, now narrowed to the two cases the queue cannot cover (slow-suite-only failures and bypass merges). The undo window there is one deploy batch, not ten minutes, and the listener still tells every PR branched after the undone change to rebase.
"I'm not urgent for it to show in production, but I am urgent to land the code."
Gera, 2026-10-01 standup, via the retro
| Who / idea | Mechanism | Wins when | Loses when | For us |
|---|---|---|---|---|
| Uber SubmitQueue EuroSys '19, 2025 update, bypass blog | A speculation tree of every pending change's pass/fail combinations (2N worst case); a model (97% accurate) picks which builds to run; changes touching disjoint build targets are independent and land in parallel; small changes bypass large ones when verified on both branches of the tree. | Builds are long (tens of minutes), volume is high, a build graph exists to prove independence. 65k changes/month. 2025: −53% CI resources, −37% p95 wait. | Builds are short enough that just running them beats predicting them. Needs a real target graph, which a Next.js monorepo does not have. | Later, if ever. The lesson to take now: the probabilistic ordering only matters once the per-PR failure rate is low. |
| Google TAP SWE book ch. 23, Memon et al., state of CI | Presubmit runs a fast subset; a change that passes it has a 95%+ chance of passing everything. Postsubmit batches changes and runs all affected tests; a failing batch is split per change to find the culprit. Reruns for flaky tests: 84% of pass→fail transitions were flakes, 1.5% of runs flaky. | Postsubmit is cheap per change because of batching; culprit finding is automated; the team accepts short red windows for speed. | Without culprit finding and revert, postsubmit is a red light nobody owns. | This is the design above: fast presubmit, batched postsubmit, culprit finding, revert. |
| Google SafeRevert paper summary | Culprit finding is 99.5% right per test but 77% per change; a model decides when a revert is safe. Historical flakiness is the strongest feature. Bad-revert rate 0.5%. | Many culprits a day, flaky history available. | Reverting on a single flaky red. Hence "retry the shard once" and "quarantined tests never trigger a revert". | Our scale needs the rules, not the model: squash = one culprit; retry once; never revert on a quarantined test. |
| Chromium CQ design doc, docs | Try jobs on a subset; failed shards retried; CLs processed out of order so one slow or flaky CL does not block others. ~100 commits/day sustained. | Flake rate is high and retries are cheap. | 15–25% of jobs were being wrongly rejected by flakes in one measured week, so retries alone did not hold. | Retry once in lanes 3–4, then quarantine; retries are a bandage, not the fix. |
| bors / homu bors-ng, Rust rollups | Batch everything approved; on failure split the batch in two and requeue both; one-at-a-time is O(N), batching O(E log N) with E failures. Rust "rollups" merge many small PRs as one. | Failures are rare and CI is slow (hours). The bisect cost log N is small next to one run. | E is large: with 1 in 7 failing, every batch bisects and you pay log N per failure. | Lane 4's bisect, capped at 16 merges so depth ≤ 4. |
| Shopify engineering blog | Predictive branch; batches of up to 8 with 3 batches testing at once; a PR is removed only after successive failures pass a tolerance (tolerance 3 at 25% flake → 0.39% false kicks); emergency bypass command. ~400 commits/day. | Flakiness is accepted as a constant and tolerated statistically. | Tolerance hides real failures for 3 cycles. | Their "master always green, always deployable" rule is ours; their tolerance is our retry-once plus quarantine. |
| GitLab merge trains docs | Each MR's pipeline includes every MR ahead; up to 20 in parallel; a failure removes the MR and restarts everything behind it. | Low failure rate; the same shape as GitHub's queue. | Exactly our Oct 1: 148 of 178 reds were "broken by a PR ahead". | Same product class we already have. |
| Mergify batching, queue modes, 2026 report | Batch size by failure rate: under 2% → 5–10; 2–5% → 3–5; over 5% → 2–3 "and fix flaky tests first". Serial, parallel (scoped by path), isolated modes. Report: 94% of merges still one PR per CI run; AI-assisted PRs failed at 1.9% vs 4.4% for non-AI. | Failure rate is known and low. | "If bisection happens more than once a day, your batch size is too large or your tests need work." | The batch-size table above is theirs, with our numbers. |
| Aviator affected targets, optimistic validation | PRs with disjoint declared targets merge independently; overlapping ones stack. Optimistic validation: a failed entry waits to see if the entries behind it (which include it) pass; if so, the failure was a flake. | Targets are computable and mostly disjoint. Flakes are the main source of kicks. | Real failures take longer to eject (depth capped at 3). | GitHub's "non-failing PRs only" switch, off, is the native version. Area lanes only if lane 2 saturates. |
| Trunk flaky, on GitHub's queue | Parallel queues from impacted targets (Bazel, Nx, Gradle); auto-quarantine: a flaky test still runs but cannot block; optimistic merging. | Many flaky tests, independent components. | Quarantine without a budget becomes a graveyard. | Quarantine with the 14-day fix-or-delete budget. |
| Graphite stack-aware queue | A whole stack of dependent PRs is validated together on one speculative branch, fast-forward merged; bisect on failure. | Developers work in stacks and merge with fast-forward. | Squash merging, which we use, breaks stacks. | Not for us; a 15-minute merge removes the need. |
| Meta predictive test selection ICSE-SEIP '19 | A model picks which tests to run per change from history; halves infra cost while catching 99.9% of faulty changes; explicitly accounts for flakiness. | Millions of tests, heavy change rate. | Needs long test history; the misses land post-merge, so it assumes a postsubmit net. | Our "affected tests" is the hand-rolled version; fine, as long as lane 3 catches what it skips. |
| Nx / Bazel affected targets Nx vs Bazel | A project graph decides which targets a change affects; Bazel needs a separate target-determinator. | Code is split into packages with declared edges. | Whole-tree checks (our drift pins) are affected by everything, so selection cannot help them; only redesign can. | Keep path-based affected tests; move whole-tree checks out of the collision class. |
| AI-agent volume, 2026 "The merge queue is the new bottleneck", Mergify report | Agents doubled PR volume into queues sized for typing speed. Recommends three tiers (pre-queue minutes, in-queue integration, post-merge heavy), quarantine, optimistic validation, and limiting an agent to 2 queue retries before escalating to a human. | — | — | The 2-kick hold and the three-tier split are taken directly. Mergify's finding that AI PRs fail less (1.9% vs 4.4%) matches our data: the bad PRs were few and repeated, not many. |
Our measurements (Oct 1–2, Central time): ~35 green main runs/day; 1,831 PRs in 30 days; 488 merges Oct 1; 50–75 queued at peaks; median PR 277 lines. Queue 24 h: 350 pass / 178 fail / 26 cancelled; 148 of 178 reds cascade; 30 own-fault = 21 queue-only + 3 flaky + 6 unsettled. Ruleset 14925316: ALLGREEN, build 20, merge 10, min 1, wait 2 min, timeout 30, squash. Oct 1 queue 9 AM–3:27 PM: 455 runs, 147 red, 128 from 5 PRs, 19 collision/stale-green; per-PR kick 14%; median 40 min in queue, p90 80; 15 merges/h on, 24/h off; 86 merges in the 15 min after off. 77 orphaned merge_group runs. Spot: 9.4% of runs lost a machine, +12.6 min; on-demand 0 of 576. CI ≈ $385/day; full suite on main ≈ $131/day; one-machine trial 6.8 vs 17.6 min (n=43). 90-day main: red 10% of hours, 367 stretches; 46% never-ran-on-PR, 23% machines, 11% live EL, 7% stale green, 7% merged red, 6% flaky; drift lists 19 stretches/month; 16.8 stretches/day in the last 4 days.
Tonight's miss, the shape this design closes: #14019 changed the renewal storage writers; the tests that depend on those writers were not "next to" the changed files, so path-based selection skipped them on the PR and they failed later in the full suite.
Sibling pages: Merge queue and deploy retro (the five attempts, the throughput model, option B), CI/CD architecture (the two-tier model, the cost decomposition), Ship in 15 minutes (PR size and review-round data), CI factory (live queue health), test speed tracker (flaky audit).
Inferred, not measured: the 5-minute fast cycle (assumes warm caches and one machine); the collision range of 1–15/day (two samples disagree; week 1 measures it); lane 4 bisect cost (assumes ≤3 red batches/day); the 60% per-push minute saving (from the trial's wall time, not a bill); "about 15 minutes end to end" (5-min lane 1 plus the review bot's ~7 min plus auto-merge).
Published sources are linked inline in the table above. GitHub's queue settings: managing a merge queue. Orphaned merge_group runs are a known GitHub behaviour with a known fix (concurrency keyed on github.ref, each queue batch has its own gh-readonly-queue/… ref): one example. REST-API bypass of "require merge queue": community discussion, Jul 2026. Ratchet files as shared append-points: one write-up, another.