Merge Speed Watch — what slows a change from "done" to "live"
48-hour live observation of every open PR · started Aug 19, runs through the morning of Aug 21 ·
LIVE — updated as findings land
Last updated: Aug 23, morning · Author: Claude (Fede's merge-speed session)
The one-paragraph summary. The pipeline itself is now fast: the median change goes from
"PR opened" to "live in production" in 57 minutes (was 74 a day ago), and once a change
is green it merges in about a minute. Nearly all remaining lost time comes from five specific traps,
documented below with receipts — and the two biggest (the big-PR test trap and the silent
auto-merge disarm) share one root cause: we verify every push instead of verifying the merge.
A decision is proposed at the bottom.
Scoreboard — 24h after the first round of fixes vs. the 24h before
| Measure | Before | After | Read |
|---|
| PR opened → change live in production (median) | 74 min | 57 min | The headline number. Target: 15 min. |
| PR opened → merged (median) | 54.5 min | 50.6 min | Modest — the median PR was already fast. |
| PRs merged in the day | 37 | 92 | 2.5× throughput, nothing left waiting on a deploy. |
| Green → merged (90th percentile) | 18 min | 9 min | When auto-merge stays armed, merging is instant. |
| Review comments per PR (average) | 5.5 | 3.7 | The founders' nit decision is holding. |
| Share of PRs taking >3 hours | 11% | 23% | Mostly good news in disguise: the old stuck backlog (four PRs aged 9–16 days) finally drained and counted its full age on the way out. |
Source: the locked pipeline-metrics.ts script (raw open→merge, no filters), snapshots Aug 19 19:37 UTC and Aug 20 19:41 UTC.
The five traps (each observed live, with receipts)
1. The big-PR test trap — decision proposed below
When a PR touches so many files that the "affected tests" selection overflows (200+ test files), the
required test check deliberately waits for the full 12-machine suite — but the full suite only runs when a
review verdict arrives. Any push after the verdict (rebases, fix-ups) waits forever: nothing
re-triggers the suite. Observed: three PRs frozen this way (later rescued by manually
re-requesting review — each rescue costing a full review round plus the full suite anyway).
The rescue day spent as much on the big test machines as the entire rest of August combined ($40 in a day
vs $78 for the month).
2. Silent auto-merge disarm
Any force-push — and even a stale "this PR has conflicts" flag that GitHub later retracts — quietly
switches auto-merge OFF, and nothing switches it back on. Green PRs then sit invisible until a person
notices. Observed: four separate incidents in 24h, including Fede's own session (the
testing-doctrine rule merged the instant it was re-armed — its whole wait was disarm, 0% pipeline).
The existing re-arm machinery only fires on the first review verdict, not on round-2+ verdicts.
3. Nobody owns the merge of workflow-file PRs (corrected diagnosis)
Correction (Aug 21): first written up as "review never fired" — that was wrong. The three green,
untouched PRs (one 3.7 days old) all change only GitHub workflow files, a class where the
bot review skips by design (GitHub forbids it) and auto-merge is policy-barred — so they need a
deliberate human merge. The trap: no one owns that step. The PRs were green and mergeable
the whole time; the process just has no "someone reads the diff and clicks" assignment. All three were
read in full and merged tonight — including, ironically, the fix for the Aug 17 stale-production incident,
which sat finished for 3.7 days.
4. Generated-file churn
Four PRs in one hour all touched the same auto-generated eval manifest. Each merge made every sibling's
copy stale → red checks on innocent PRs → main itself briefly red → two extra "regenerate" commits to fix.
The file is pinned per-branch but describes whole-repo state — it should be regenerated once at merge time,
not raced on by every branch.
5. Dropped work in a stacked-branch rebase (caught & recovered)
A fully-reviewed change (4 review rounds — the "final reply must stand alone" fix that stops Clara claiming
she sent a link she didn't) had been merged into another PR's branch, then silently thrown away when
that branch was rebased. It existed nowhere in production for ~5.5 hours. The reviewer's re-check flagged it;
the exact reviewed commit was recovered and re-landed as its own PR — open to merged in 15 minutes, hands-off.
Watch item: stacked PRs + rebases can silently delete merged work; the review's "re-verify after rebase" habit
is currently the only net.
Where an open PR's time actually goes (756 PR-hours, measured in 10-minute snapshots over 24h)
| Share | State | Read |
|---|
| 42% | Deliberately held (labeled hold-for-review) | By choice — mostly Gera's grading-playground stack. Excluded from stall math. |
| 26% | Green but idle — nothing running, nothing armed, waiting for a person | The waste. 201 hours of finished work sitting. Top offenders: the ownerless workflow-file PRs (trap 3) and PRs whose authors moved on. |
| 20% | Red check, waiting for the author to fix | Author response gaps — a session/human attention problem, not machinery. |
| 5% | Green, armed, in the merge queue | Healthy. |
| 4% | Checks actually running | The pipeline computes for 4% of the time. Every speed problem we have is a waiting problem, not a compute problem. |
| 1% | Draft | — |
Where the GitHub Actions money goes (August: ~$1,465 gross, ~$800 billed after credits)
| Share | What | Caught in the last 60 pushes |
|---|
| ~61% | The per-push CI battery (affected tests, type check, lint, six guard jobs, simulate) — re-run on every push, ~6 pushes per PR | 2 type errors. Everything else: zero real catches — authors (Claude sessions) already verify locally before pushing. |
| ~34% | 89 small helper workflows — the largest boots a runner on every PR comment just to check if it says "@claude" (1 in 5 of all runs) | Glue, not verification. 12,300 runs in 2 days. |
| ~5% | The 12-machine full test suite (escalation / nightly / main only) | Legit — but manual big-PR rescues doubled its monthly cost in one day. |
Decision: how to fix the big-PR trap (and most of the cost)
DECIDED Aug 21 (evening): keep watching — Fede chose not to build any of the
options yet. Rationale recorded: cost cut #1's clean-day verdict lands Aug 22, and Gera's
review-on-synchronize change (#6053, held for Fede) would already dissolve the disarm trap and
un-deadlock big PRs by a different route. The options below stay on file; revisit if the Aug 22+ numbers
miss the ≤$80/day target or the traps recur.
Option 1 — Big PRs run the full suite on every push
Simplest possible rule; no stuck states. Rejected by Fede Aug 19 — too expensive
(every push on a big PR = a $1.20 full-suite run).
Option 2 — The patch: full suite auto-runs on pushes that land after a verdict
One-line-ish change to the escalation rule. Kills the deadlock exactly where it bites, costs ~22¢ per
post-verdict push — cheaper than today's manual rescues (which buy a full review round + the suite
anyway). Leaves everything else (cost, disarms, generated-file churn) as-is. Low risk, ships in an hour.
Option 3 — The redesign: verify the merge, not every push RECOMMENDED
Per push: nothing (or a 1-minute incremental type check — the only per-push job that caught anything).
At merge: the entire battery, once, on the true merge result — full suite, guard pins, lint,
security scan, and regeneration of generated files.
This one change kills traps 1, 2-partially, and 4 outright: no affected-vs-full modes, no
escalation machinery, no stuck test checks, no per-branch generated-file races — and cuts the biggest cost
bucket ~5–6× (one battery per PR instead of ~6). Honest tradeoff: a genuinely broken PR bounces at the merge
queue (~10 min round trip) instead of failing fast on push — at the measured 1–3% failure rate, a rare small
delay. Bigger build: a few days, staged.
Option 4 — Both, staged: ship the patch now, build the redesign behind it
The patch un-deadlocks big PRs today; the redesign then deletes the machinery the patch lives in.
Right choice if the redesign timeline matters less than never seeing the trap again this week.
Recommendation: Option 3 (or Option 4 if you want the bleeding stopped this week.)
Cost reality-check (verified Aug 21): the easy trigger-level levers are already
pulled — every heavy workflow already cancels superseded runs on a new push, and the comment-triggered
runner boots were measured at ~$0 each. An earlier draft of this doc claimed a 30–40% cut from those; that
was wrong and is retracted. With spend at $145 → $170/day (all grant credit — $0 cash today, but the
$10k credit is ~2 months deep at this pace, then it's ~$5k/month real), the only lever that halves
the bill is structural: stop paying for the full battery ~6× per PR. That is Option 3. The one surviving
no-tradeoff fix: re-arm auto-merge on every verdict, not just round 1 (fixes trap 2; costs nothing).
Trap 3 ownership — RESOLVED Aug 21 (evening): the daily digest is built (PR #6055,
workflow-pr-digest.yml). Once a day, #alerts gets one plain-English message listing every
finished, green workflow-only PR waiting for a human read-and-merge, with ages; quiet days post nothing.
It never merges anything itself. The watch session keeps clearing them opportunistically while it runs;
the digest is the durable backstop that survives any machine shutdown.
Live log — added as found
Aug 23, morning — cost verdict: target HIT on the first clean day
Aug 22 closed at $54.45 — under the ≤$80 target and under the ~$63 pre-spike norm.
The four-day line: $168.54 → $137.23 →
$54.45, a 68% drop from the peak. One honest
caveat: Aug 22 was a Saturday, so PR volume was lighter than a weekday — the cut is clearly working, but
the confirmation under full load is Monday Aug 25's number. (Aug 23, also a weekend day, is at $29.12 by
morning.)
Read: cost cut #1 (guard pins post-merge only) plus the quieter board appear
sufficient; the shelved "verify the merge, not every push" redesign stays shelved unless weekday spend
climbs back above target.
Trap-3 digest: first scheduled run fired Aug 22 15:53 UTC and
behaved exactly as designed — no finished automation-only PRs were waiting, so it posted nothing.
Aug 22, morning — cost verdict, day 1 after the guards cut
Aug 21 closed at $137.23 — down 19% from the $168.54 baseline, but well short of the
predicted ~$115–125. The mid-day projection missed because the evening was unusually busy (the digest
workflow's own PR ran three review rounds, several reviews were re-fired on stalled PRs, and Gera pushed a
fresh batch) — and Aug 21 wasn't a clean day anyway (machine down all morning suppressed pushes, then the
backlog burst in the evening).
Aug 22 is the honest test and it has started well: $28.96 by 9am
Denver (Aug 21 was already ~$45–50 by this hour on its way to $137). Too early to call ≤$80;
the full-day number lands tomorrow morning.
Speed: no stalls this morning — the two PRs
caught green-with-no-review in overnight sweeps (#5807, #6080) were both handled (one turned out to be a
held-for-Fede diff, the other merged after a re-fired review). The trap-3 digest workflow is live on main;
first scheduled post 9:47am Denver.
Aug 21, evening — machine restart; watch moves to a lighter cadence
The Mac was shut down mid-day and is back up. The GitHub-side machinery (auto-merge,
re-arm, reviews, alerts) ran fine on its own the whole time — nothing got stuck, and no ownerless
workflow-file PRs accumulated. The heavy local collectors (10-minute snapshots, webhook feed) are
not being restarted: the 48-hour observation is complete and its data is archived. The watch
continues on a lighter cadence — a periodic stall sweep plus the daily cost check.
Cost: Aug 20 closed at
$168.54 (the "$170" baseline, now exact). Aug 21 is
at $98.63 by evening — on track for the predicted ~$115–125 after cost cut #1. Aug 22 is the first clean
full day and the real verdict (target ≤$80 gross).
New development that bears on the decision below: Gera opened a change ("review on
synchronize", #6053) that makes the review bot automatically re-review after every fix-push, with evidence
that 45 of 48 recent watchdog pages trace to reviews never re-firing after a push. If it lands, it largely
dissolves trap 2 (every push gets a fresh verdict, which re-arms auto-merge) and un-deadlocks trap 1 by a
different route (a verdict on the new head re-triggers the big-PR full suite). It is cost-bounded per its
analysis (push bursts cancel their own older runs before any work happens), but it does mean more
review/full-suite activity per PR — it leans
against the "verify only at merge" redesign's premise,
or at least changes its starting point. It is correctly held for Fede (touches security-relevant workflow
files, and flips an ADR from deferred to adopted). Read it before picking an option below.
Aug 21, 13:45 UTC — morning update
Cost: first partial day after the guards cut: $72.53 by mid-day, tracking ≈$115–125 vs
$170 yesterday — consistent with the predicted ~⅓ cut; the first full clean day (Aug 22) is the real verdict.
Guards confirmed skipping on every PR push.
Speed: 27 PRs merged overnight; green→merged
median holds at ~6 seconds. The trailing 24h median (67 min) is polluted by the backlog drain — the
days-old PRs merged during the sweep count their full ages (p90 shows a 6-day-old PR).
Trap 3
recurred twice in 12 hours (two new green workflow-only PRs sat ownerless; both read in full and
merged — a nightly-harness credentials fix and a stop-paging-over-successful-reviews fix). The class needs a
standing owner or digest — proposal added to the decision section.
Aug 21, 04:15 UTC
Cost cut #1 SHIPPED (Fede's directive: halve the bill): guard pins no longer run on PR
pushes — post-merge only, where they already ran and already feed the red-main alert. They were 57% of all
per-push CI minutes with zero author-bug catches in 60 pushes. Expected effect ≈ one third of total Actions
spend ($160 → ~$105-110/day); tomorrow's billing shows the real number. The remaining stretch to half is the
Option 3 decision above. Also verified & retracted earlier: superseded-run cancellation was already in
place everywhere, so no savings existed there.
Aug 21, 03:30 UTC
Trap 3 re-diagnosed (see corrected section above) and cleared: all three ownerless workflow-file PRs read in
full and merged — the Aug-17 stale-prod fix (3.7 days old), the mis-pinned-review repair (which itself fixes a
machinery bug from this report), and the touch-budget alert-fatigue fix. Phase-decomposition of the full 24h
snapshot data added: checks run 4% of the time; green-idle is 26%.
Aug 21, 01:50 UTC
Gera's PRs brought into active watch (Fede's request). Found trap #3 (reviews never fired: #5975, #5967,
#5827 — re-fired tonight); #5981 waits on Gera's answer to review suggestions; two PRs have real merge
conflicts only he can resolve; nine are deliberately held (grading-playground stack) and excluded from stall math.
Aug 20, 14:13 UTC
The application-link PR merged (Fede's explicit click — correctly held by the permissions-file hard floor).
Full app-link family now live.
Aug 20, 03:40 UTC
Trap #5 recovery: the dropped "final reply must stand alone" work re-landed (15 min, hands-off).
Aug 20, 03:30 UTC
The two deadlocked big PRs (label-quality sweep, topic-labeler corpus) merged after rescue. Main briefly red
from trap #4; healed by peer sessions within the hour.
Data sources: locked pipeline-metrics.ts · 10-minute lifecycle
snapshots of every open PR (48h) · GitHub billing usage API · per-job analysis of the last 60 CI runs ·
incident receipts in the session log. Overnight collection continues; morning update lands ~7:30am Denver.