Why GitHub's merge queue failed five times in eleven weeks, why deploys choked once it went off, and the order of work to get stable.
2026-10-01 · times in Central · Proposed — pending review
On this page
Every attempt broke for a different surface reason, but they all share one cause: we switched the queue on before the PRs going into it reliably passed it. Each time, the queue was the first place some checks ever ran, and we kept changing CI underneath it while it carried the whole team's merges.
A queue multiplies failure. One failed PR throws away the work of every PR stacked behind it. Today about 1 in 7 PRs got kicked out, so the line could clear only about 24–29 PRs an hour. We arrive at about 20–24 an hour. The line grew to 72 and nobody could merge.
Clashes between PRs are real but a minority: about 1 in 6 red-main episodes in the last 30 days. Most red main is machines, tests that never ran on the PR, or one bad PR.
Turning the queue off moved the choke point; it did not remove it. Main went red 4 minutes later and deploys stopped. Same cause: tests that never ran on the PR.
Recommendation (option B): stable first. Queue stays off. Every check that can turn main red runs on the PR. A red main is fixed within 30 minutes by one owner. CI stops changing under the team. Cost and speed come after.
GitHub's own change history for the "Require CI" rule (every on/off and setting, to the second), the CI decision log, the CI/CD design page and the "ship in 15 minutes" page, the factory dashboard, the July decision record that retired the first queue, PR bodies, today's standup transcript, the CI lane's session transcripts, and the queue, main and cost measurements taken today. Anything worked out rather than measured is marked (inferred). Full list in the appendix.
"I want a retro document first. I don't want more hacks." … "and priorities to get things stable right now. Number one."
Fede, 2026-10-01
Stability is number one. A red main stops merges, deploys and everyone's PRs at once, so nothing else pays off until it holds.
(a) Every test that can turn main red runs on the PR before it merges. That is the full bookkeeping set (tests that keep shared lists, like routes and fields, in sync) plus the safety checks. About 4 minutes in parallel, cents per push (inferred). Today the PR runs only the subset picked by which files changed: about 310 of 1,044 guard files. The queue and main ran the rest.
(b) No stale greens. A PR whose full checks ran on an old copy of main re-checks against current main before it merges.
(c) Machines don't die mid-run. Machines and deploy steps caused 23% of red stretches over 90 days (18% of red hours), and about 16% since Sep 13. An earlier 30-day sample said 55%, but it was skewed by the Sep 8–12 machine incident (see section 3). Infra reliability is part of keeping main green, not a cost question.
Evidence: today's breaks at 3:31 PM and about 4 PM. In the 4 PM break, three culprits never ran the failing test. The fourth had passed the full set two days earlier and merged with no re-check. The 3:31 PM culprit's test never ran on its PR either.
We already have the one-driver rule: one session owns a red main, everyone else keeps working. Today main was red from 3:31 PM to at least 4:20 PM. Deploys stalled 47+ minutes with 119 changes waiting. About 37 open PRs showed red only because main was red.
CI changes go out only in a set window, each after a trial run. 29 CI changes merged during the last queue attempt. The machine type flipped 3 times in 20 hours.
Cost is not down yet: about $385/day measured at 1 PM, flat vs yesterday. The target is under $65.
One-machine trial. Since this morning, 1 in 4 PR pushes also runs all checks on one machine, silently, gating nothing. In 43 samples it finished in a median 6.8 min vs 17.6 min for today's multi-machine setup, faster in 40 of 43. Not proven yet: its list-test step failed in every sample, probably because main was red (inferred). It needs a clean re-check before any switch. Full comparison due 2026-10-02 around noon.
Why B: a stale list entry does not change what customers get (inferred), yet today it held 119 changes out of production. This is a rule change, not a code hack. It waits on Fede's yes.
Main was red for about 234 hours in the last 90 days, about 10% of the time, across 367 separate red stretches. Before the merge queue was retired on Jul 22 it was red about 1% of the time. After that it was red about 11%. Red stretches per day went from 0.4 to 3.4, then 7.6 in mid-September and 16.8 in the last four days.
| Why main went red | Share of red stretches | Share of red hours | What it means |
|---|---|---|---|
| The test that broke main never ran before merge | 46% | 57% | The PR ran a subset of the tests. The one that broke only runs on main. |
| Machines and deploy steps | 23% | 18% | Runner shutdowns mid-run and Vercel deploy errors |
| Stale green | 7% | 11% | The PR passed days ago, main moved on, and it merged without re-testing |
| Live voice-agent settings changed | 11% | 5% | Tests that read ElevenLabs's live settings. Main goes red with no code change. |
| Merged while red | 7% | 5% | Admin merges, and a required check that does not check the code |
| Flaky | 6% | 4% | Passed on the PR, failed on main, and no fix was needed |
This is a class problem, not a few bad tests. It takes 73 different tests to cover 80% of the red hours. The repeat offenders are mostly bookkeeping lists that a new page or setting has to be added to: the sensor registry (16 times), the route manifest (11), eval model pins (8), the user route inventory (7), and the single-writer list (7).
| Fix | Red stretches removed per month | Red hours removed per month |
|---|---|---|
| Run every test that can break main on the PR, against current main | 35 | 32 |
| Bookkeeping lists update themselves instead of failing the build | 19 | 12 |
| Machines that don't die mid-run (spot quota, guard pins on 4+ CPUs) | 25 | 9 |
| Re-test a stale green before it merges | 8 | 8.5 |
| Vercel deploy errors stop turning CI red | 3 | 5 |
| Live voice-agent tests move off the blocking path and alert instead | 13 | 4 |
| Close the merge-while-red path | 8 | 4 |
| Quarantine flaky tests (Gera's audit) | 8 | 3 |
These are estimates that assume each fix is fully effective. They overlap, so don't add them up.
Recommended order.
Flaky tests come from Gera's audit (test speed tracker). Nothing here is built yet, because CI changes are frozen until Fede and Gera pick the order.
Correction: an earlier version of this page said machines caused 55% of red main. That figure came from a 30-day sample dominated by the Sep 8–12 machine incident, which was fixed Sep 12. The full 90-day count says machines and deploy steps caused 23% of red stretches. Since Sep 13 the figure is about 16%.
Method: every red stretch from Jul 3 to Oct 1 was classified, with no sampling. The culprit is the PR behind the first red commit. "Never ran before merge" versus "stale green" was read from PR logs for 194 stretches. "Flaky" is inferred. Red hours include overnight gaps with no pushes, so overnight stretches count high. Data: scratchpad deep-ci/main (episodes, pareto, fixes, repeat offenders).
A merge queue is a GitHub feature that lines PRs up and tests each one combined with everything ahead of it before it lands. If a PR fails, it is kicked out and every PR behind it is retested without it.
| When | What we changed | What happened | Why it failed |
|---|---|---|---|
| 1. Jul 14 – Jul 22 8 days |
The queue became the main test gate. PRs ran a small slice of tests; the full suite ran only in the queue. One PR per batch was allowed. | It never batched: 66 queue runs for 63 PRs in a day. A green PR waited for a second full run (4–13 min). Failures were flaky tests on two bad days (18 and 24 failures), each restarting everyone behind. About a quarter of the CI bill. | Too little traffic to batch: PRs arrived 5–12 minutes apart, slower than the queue emptied. Its catches were flaky-test storms (inferred): 104 red in 615 runs, no one-off cross-PR catches found. Retired; the full suite moved to after merge. |
| 2. Sep 13, 10:49–11:31 PM 42 min |
"Take two": real batching (at least 2 PRs, 3-minute wait), full suite in the queue. | CI ran and passed on the queue's test commits, but the checks GitHub requires never reported there, so no PR could pass. Four PRs sat about an hour. | Plumbing: the queue's checks were not the same as the PR's required checks. Gera turned it off by hand. |
| 3. Sep 13, 11:48 PM – 12:23 AM 35 min |
Switched back on the same night. | Same stall. The fix for the reporting merged about 80 minutes after it went off. | Same cause. Gera removed it again; Fede ruled the next day that it stays off, and the code was reverted. |
| 4. Sep 30, 5:17–6:38 PM 81 min |
After the CI/CD design review: queue on, batches of 5, then 10, in the middle of a backlog. | The first batch lost a cheap cloud machine that Amazon took back, so the whole batch failed and re-queued (about 45 min lost). 4 merges in the 5–6 PM hour. 97 PRs merged from 6:30 to 7:10 PM, almost all after it went off. | One machine loss killed a batch of up to 10 PRs, at a time when we needed to catch up. Fede: "is the queue even worth having right now that we are trying to go faster and catch up". |
| 5. Oct 1, 5:49 AM – 3:27 PM 9.6 hours |
The new approach: only the fast checks in the queue (full suite after merge), every required check reports in the queue, full-price machines, batches raised 5 → 10 → 20. | 39% of queue runs failed. Median 40 min in the queue, 1 in 10 waited over 80. Line grew from 47 to 72. Merges fell to about 15 an hour, down from about 24 overnight. | A pile-up, detailed in the next box. Off on Fede's "We need emergency action." 22 PRs merged within 3 minutes. |
After "stop at the first failure" landed: 27 runs passed, 0 failed, 5 cancelled, in the 24 minutes before the queue went off. Too short to credit it alone; the broken CI change and the re-queued PR were already out by then (inferred).
Today, end to end: PR opened → merged median 73 min (1 in 10 over 9 hours); merged → live median 79 min.
Two earlier write-ups each proposed a new approach. Both read the problem as "the queue is set up wrong," not "PRs arrive unready."
"July's queue ran one full suite per PR by construction … The new config batches 2–3 (up to 5) PRs per run after a ~3-minute wait, fixing the exact 'never batched' root cause."
"Merge queue, take two", ship-in-15-minutes page, Sep 13. It failed that night on a different cause: checks not reporting.
"Fix: test each PR against the latest main before it merges (a merge queue), then deploy that exact result with a small canary and automatic rollback."
CI/CD design review, Sep 30, "Decided: merge queue yes (after the backlog is caught up)".
The Sep 30 plan had the right order, and we skipped it. Its step 3 was "the bookkeeping tests always run on PRs, never skipped." Its step 5 was "merge queue on, about a week in shadow first." Instead, the full bookkeeping set was moved off PRs that same evening and the next morning. The queue went on the same afternoon, with no shadow week, during the backlog.
"It seems like the bottleneck moved to the deploy stage… I moved the merge queue and now the choke point is on the next step."
Fede, 2026-10-01
Lesson: the queue was hiding the same root cause. Without it, the failures land on main instead of in the queue. The place changed; the cause did not.
The queue was always the first place some checks ran, and it was switched on live, with no trial week. In a queue, every failure is paid by everyone behind it. So anything that leaks into the queue gets multiplied: a check gap, a flaky test, a lost machine, a bad CI edit. Without a queue, the same thing costs one PR.
| Candidate | Seen in | Verdict |
|---|---|---|
| Checks that run only in the queue | 1, 2, 3, 5 | root July: full suite only in the queue. Sep 13: required checks never reported. Oct 1: the bookkeeping tests, including the script-safety check. |
| High failure rate, with everyone behind restarting | 1, 4, 5 | the multiplier Turns any failure into lost hours for the whole line (see the math). |
| Our own CI changes while the queue was live | 4, 5 | root We debugged and rebuilt CI on the same line the team merges through. |
| Arrival rate vs capacity | 1 (too low), 5 (too high) | flipped July: a PR every 5–12 minutes, nothing to batch. Now about 20 an hour, which leaves no slack for failures. |
| Cheap machines that get taken back | 4, 5 | trigger Killed attempt 4. On Oct 1, after the switch back, about 9% of runs lost a machine and ran about 12.6 min longer. They showed up as cancelled runs, not failures. |
| Main red | 5 | amplifier With a queue, a red main stops everyone, and the red-main fix path (drop the rule) flushed the line. |
| Agents re-queuing kicked PRs, line-jumping | 5 | amplifier One unfixed PR caused 9 of 11 failures in one stretch. One jump voided 9 builds. |
| Slow, heavy queue runs | 1, 5 | contributor About 12 min per run. The main build-and-typecheck job took 9 min in the queue vs 3.4 on a PR (an extra build and cold caches). |
GitHub builds the line as nested stacks: PR 1; PR 1+2; PR 1+2+3; and so on. If PR 3 fails, every stack from 3 up is thrown away and rebuilt. So the number of PRs that land per cycle is capped by how many pass in a row, not by how many builds we pay for.
On paper: 10 builds ÷ 12 min = 50 PRs an hour. With 1 in 7 PRs kicked (today's 14%), on average only about 6 pass in a row. Each 12-minute cycle lands about 5, so about 24 an hour, and doubling the builds to 20 only reaches about 29. That is why raising builds from 10 to 20 at 2:55 PM did not help.
| PRs kicked (p) | PRs/hour, 10 builds | PRs/hour, 20 builds | vs ~20–24/hour arriving |
|---|---|---|---|
| 2% | 45 | 81 | 2× headroom |
| 3% | 42 | 74 | 2× headroom |
| 5% | 38 | 61 | thin in bursts |
| 14% (Oct 1) | 24 | 29 | line grows |
| 25% | 14 | 15 | collapse |
Run time 12 min. These are best-case ceilings (inferred): they ignore lost machines, line-jumps and a kicked PR's own rework, all of which happened. A line that clears barely faster than it fills grows without limit in any burst. The 39% figure is per queue run, not per PR. It means 4 in 10 runs produced nothing; a midday cost check found 57–65% of queue machine spend going to failed runs.
"Tests PRs against each other makes sense. But did we identify that as a common pain point / bottleneck in real ci runs? Did we sample last 30 days to see if that's really saving the issues."
Fede, 2026-10-01
Answer: clashes between PRs are real but a minority, about 1 in 6 red-main episodes. Most red main is machines, tests that never ran on the PR, or one bad PR. The queue does catch stale greens, but today it spent most of its runs on 5 bad PRs. This supports option B: close the PR-side gaps and add a stale-green re-check first.
A collision means two PRs that each pass alone but break main together. A stale green is the same thing over time: a PR passed on an old main and broke on today's. These are the only failures a queue catches that PR checks cannot. All numbers below are samples and partly (inferred).
| Sample | Collision or stale green | Test never ran on the PR | Flaky test | Machines and outside services | Other |
|---|---|---|---|---|---|
| Red main, Sep 1–Oct 1: 958 red builds in 264 episodes (130 analysed) | 15% | 12% | 7% | 55% | 2% red on the PR and merged anyway; 10% unknown. Machines shut down mid-run in 54 episodes, the live voice agent changed in 8, deploy steps in 7 (this sample is skewed by the Sep 8–12 machine incident; the full 90-day count is 23% infra, see section 3) |
| Oct 1 queue, 9 AM–3:27 PM: 455 runs, 147 red | 19 runs | — | — | 0 red | 128 red runs came from 5 bad PRs failing every batch they joined: two real-database test problems (46 and 44 runs), a marker check (18), our own CI change (12), a workflow pin (8). Lost machines showed up as 16 cancelled runs, not failures |
| Oct 1 main breaks after the queue went off (5 culprits) | 1 | 4 | — | — | 3:31 PM: a CI-settings change whose pinned test never ran on its PR. ~4 PM: 3 PRs that never ran the list tests, plus 1 stale green |
| Sep 30 review: the "25 of 34" failed main runs (one day) | 0 | — | — | 3 runs | 21 runs came from 2 PRs whose list test was already red on their own PR and merged anyway. 3 were the live voice agent changing; 1 untraced. These were runs, not causes |
| July queue: 615 runs, 104 red | 0 | — | most (inferred) | — | The same test failed in up to 32 runs: flaky-test storms (inferred). No one-off catches found. The older count said 434 runs |
What it means for the plan: machines and outside services are the largest share of red main, and machine shutdowns alone are the single largest cause. So infra reliability now sits under "main stays green" in section 2. A queue stays a maybe for later, not a fix for now.
| Condition | Needed | Oct 1, measured |
|---|---|---|
| Same checks on the PR as in the queue | No check runs only in the queue | The full bookkeeping set ran only there |
| PRs kicked from the line | Under 3% (about 2× headroom at 10 builds) | 14% |
| Queue runs failing | Under 5% | 39% |
| One queue run | 10 min or less, stopping at the first failure | About 12 min; stop-early only from 3:03 PM |
| Machines lost mid-run | Under 1% of runs | 9% on cheap machines; 0% on full-price |
| Main green | 95%+ of the workday; red-main fixes land without touching the queue rule | Red about 2 hours; the rule drop flushed 70 PRs |
| CI changes while the queue is on | None outside a set window, each tried in a trial run first | 29 in under 10 hours |
| Agent behavior | A kicked PR re-enters only with a new commit; line-jumps only for red-main fixes | Both broken |
| Trial first | One week measuring all of the above with nothing waiting on it | None of the five attempts had one |
| Option | What it buys | What it costs |
|---|---|---|
| 1. No queue (today): merge on green, heavy tests after merge, automatic revert of a single culprit, deploys held while main is red | Merges never wait. Matches Gera's priority: "I'm not urgent for it to show in production, but I am urgent to land the code." | Main goes red whenever the PR ran fewer checks than main. 4 minutes after the queue went off, a PR that changed CI machine settings merged on green and broke main: a pinned test the PR never ran. Overnight before the last attempt, 172 of 187 main runs failed. Gera's own flaw: "Bad code reaches main. We're all gonna pull it." |
| 2. Require up-to-date branches, no queue | Every PR is tested on the latest main. Catches collisions exactly. | Main moves every ~3 minutes; a PR's checks take ~11. PRs never catch up. Measured in August at about 30 merges a day, a fraction of today's 400–500: 39% of all commits were catch-up syncs, about 44 hours of re-testing unchanged code in two days, one PR synced 15 times. |
| 3. Queue with small batches, or "head-only" checking (only the last stack in a batch must pass) | Smaller blast radius per failure; fewer runs. | Does not fix the failure rate. At 14% kicked, batches of 4 clear about 14 PRs an hour, below arrivals (inferred). Head-only lets a batch land on one result, so when it fails, finding the culprit costs more. Not tried here. |
| 4. Merge train with automatic bisect and revert: merge on green, test main every few minutes, find the culprit and revert it | Merges never wait; heavy tests run once per train. Most parts exist: the after-merge lane, single-culprit revert, the deploy hold. | Main is red between a bad merge and its revert. A flaky test causes wrong reverts. Whether reverts happen at all is still an open question to Fede. This is not the Zillow behavior Fede described ("you just get kicked out of the queue"): that is fail-fast inside a queue. |
| 5. Run the queue's full check set on the PR, so a queue (if any) only re-checks | Removes the biggest failure class from both worlds. It would have caught Oct 1's queue-only failures before they entered the line, and the 3:31 PM break after it. | A slightly longer PR lane: the bookkeeping set runs as 6 parallel ~4-min pieces, alongside checks that already take 3–9 min. About 4 cents a push on cheap machines (inferred). Does not catch true collisions. |
| 6. Gera's idea: own tests first, then a capped queue. "Does the PR pass its own section's tests; if so, add to merge queue… limit it to 5 PRs at a time, every 10 minutes." | A steady, predictable pace. Small batches limit the damage per failure. Only PRs that passed their own tests enter. | Works only when fewer than about 3 in 100 PRs get kicked. At today's 1 in 7, a 5-PR batch passes about half the time (0.865 ≈ 0.47) and clears about 15–20 an hour, below the 20–24 arriving. Under 3%, it clears about 25–28 an hour. "Its own section's tests" is exactly today's gap: the section tests miss the list tests. One bad PR delays four good ones by 10+ minutes, plus the time to find which one. |
Why B: it puts stability first and fixes the cause shared by all five attempts and both of today's main breaks. None of it needs a queue, and none of it is a hack. Gera's capped queue (option 6) only works after step 1 brings kicks under 3 in 100. C repeats attempt 5 at a 14% kick rate. D was measured as a treadmill in August at a fraction of today's volume. E keeps main red between a bad merge and its revert, which is the thing Gera flagged.
| Version | Time (CT) | Actor | State |
|---|---|---|---|
| 43073673 | Jul 14 12:13 PM | fede-propflow | ON: ALLGREEN, build 5, merge 5, min 1, wait 5, timeout 30 |
| 43115357–43610155 | Jul 14–19 | both | timeout 15/30/45, wait 5→10 |
| 44072984 | Jul 22 1:15 AM | gera-propflow | OFF |
| 49589307 | Sep 13 10:49 PM | fede-propflow | ON: ALLGREEN, build 5, merge 5, min 2, wait 3, timeout 60 |
| 49591082 | Sep 13 11:31 PM | gera-propflow | OFF |
| 49591753 | Sep 13 11:48 PM | fede-propflow | ON (same settings) |
| 49593303 | Sep 14 12:23 AM | gera-propflow | OFF |
| 51497955 | Sep 30 5:17 PM | fede-propflow | ON: build 5, merge 5, min 1, wait 5, timeout 90 |
| 51498771 / 51498782 | Sep 30 5:27:49 / 5:28:03 PM | fede-propflow | 14-second OFF/ON (admin merge, inferred) |
| 51501821 | Sep 30 6:17 PM | fede-propflow | build/merge 10, wait 3 |
| 51502749 | Sep 30 6:34 PM | fede-propflow | build/merge 5 |
| 51502913 | Sep 30 6:38 PM | fede-propflow | OFF |
| 51556813 | Oct 1 5:49 AM | fede-propflow | ON: build 5, merge 5, min 1, wait 2, timeout 30 |
| 51566658 | Oct 1 7:20 AM | fede-propflow | build/merge 10 |
| 51619163 | Oct 1 2:55 PM | fede-propflow | build 20, merge 10 |
| 51620073 / 51620082 | Oct 1 3:03:15 / 3:03:21 PM | fede-propflow | 6-second OFF/ON to admin-merge #14934; silently flushed the queue |
| 51622689 | Oct 1 3:27 PM | fede-propflow | OFF |
strict_required_status_checks_policy is false in every version from Jul 3 on. The August "strict: true" reading on the delivery page came from classic branch protection, not this ruleset (inferred).
docs/adr/0107-deploy-gate-relocation-retire-merge-queue.md), follow-up #4444.150caf5f04 concluded success but none of the required contexts were posted. Reverted by #8485 (Fede 2026-09-14).src/__tests__/ci-cost-routing.test.ts was stale, and the PR lane runs affected tests only. The fix was folded into another PR. ~4 PM CT: second break in the route-inventory and legacy-field list tests (three culprits path-selected out of the guard set, one stale green whose full guard run was two days old). Deploys held from 3:28 PM CT; factory dashboard 4:17 PM CT "On main 55, Deploy batch 54"; 119 changes waiting, ~37 open PRs red from main's red.spot-int/report.md, 9:04 AM–2:11 PM CT): on-demand era 576 critical-job executions, 0 interruptions; spot era 2,222, 30 interruptions; 24 of 256 runs (9.4%) affected, +12.6 min median.gh api repos/PropFlow-Technologies/propflowai/rulesets/14925316/history and each /history/<version>.git log -S merge_group on origin/main; docs/runbooks/ci-and-deploy.md (pre-enqueue paragraph); ADR-0107.~/.claude/logs/ci-decisions.log (Sep 30 night to Oct 1, 3:36 PM CT).