Why Shipping Got Slow Again

Deep dive, Aug 18 2026 · follow-up to Lead Time: The Top Moves (Aug 10–11) · status: Proposed — pending Fede's review

TL;DR — three separate things, one now fixed, one real, one an illusion.

The early-August cliff is over. The week of Aug 3 a typical change took 4+ hours to merge and 55% took over 3 hours. The Aug 10–11 fixes (killing the re-prove-everything-on-every-merge treadmill, making advisory review notes non-blocking, faster build machines) worked: the treadmill is verifiably still off today — zero catch-up-to-main commits in the 40 most recent merged changes.
What's still real: the test pipeline itself doubled. Getting a green check went from ~4 min (July) to ~8 min now, worst cases ~19 min. Cause: a cluster of CI additions Aug 8–11 grew each change's check run from ~6 jobs to ~15, and the expensive run-all-2,000-tests path now fires on 30% of runs instead of 12% — despite a change meant to reduce exactly that. More jobs also means more time queuing for machines (~4× longer waits).
"Reviews ballooning" is mostly a broken gauge. On recent changes the reviewer posts ~9 review events per change, but authors only push 3–4 times, and just 13 of 368 review events found anything blocking. One change collected 28 review events over 5 pushes with zero findings. The extra events are duplicate/phantom posts by the review machinery, not extra rounds of real work — the metric counts them, so it climbs while actual behavior is fine.

Typical change today: ~49 min from opened to merged (July: ~25–30) plus ~18–20 min for the deploy to promote. The gap to July is mostly the CI doubling.

The numbers

Opened → merged, by week

Week	Changes merged	Median	Over 3 hours
Jul 6–12	497	29 min	14%
Jul 13–19	515	30 min	17%
Jul 20–26	482	18 min	14%
Jul 27–Aug 2	561	27 min	18%
Aug 3–9 the cliff	236	4.2 h	55%
Aug 10–16 recovering	173	54 min	34%
Aug 17–18 still 2× July	29	49 min	17%

Raw open→merge on every merged change, the one locked definition (Aug 10). Merge volume also fell from ~70/day (July) to ~10–30/day — not diagnosed here; flagged at the end.

Time to a green check (the test pipeline)

Window	Median	Worst 10%	Jobs per run (median)	Full-suite fires on
Jul 13–19	4.2 min	9.2 min	6	12.5% of runs
Aug 10–16	5.2 min	11.3 min	—	—
Aug 17–18	7.8 min	18.8 min	15	30% of runs

Both halves grew: the jobs run longer and they wait ~4× longer for a machine — because 2.5× more jobs per change now compete for the same shared runner capacity. It is not classic runner starvation from outside traffic; it's self-inflicted fan-out. The additions, all Aug 8–11: the always-on guard checks split into 6 parallel jobs per change (meant as a speedup; net effect was more queue contention), a new database-safety gate, a new debounce step, and heavier type-checking. Separately, the "escalate to the full 12-shard test suite" trigger now fires 2.4× as often as July even though an Aug 9 change was meant to reduce re-firing — why is unverified and worth its own look.

Review rounds: what the climbing metric actually measures

Dashboard says rounds per change went 6.7 → 9.9 → 11.4 (30-day → 7-day → last 24 h). Ground truth from the 40 most recent merged changes:

Review events per change: mean 9.2. Author pushes per change: median 3. The reviewer machinery posts ~2–3 events per push (two bot identities posting on the same push, empty-body inline-comment events, verdict re-posts).
Blocking findings: 13 out of 368 review events (3.5%). Extreme case: one change drew 28 review events over 5 pushes, zero findings.
So current inflation is phantom events, not extra review work. The Aug 11 audit saw this ("the '17 rounds' were really ~5 real reviews plus phantom events") and halved duplicate verdicts — but the double-posting persists and the rounds metric still counts raw events.

History for the record: during the Aug 3–9 cliff the churn was real — the worst change collected 22 review rounds and 59% of its commits were pure catch-up-to-main syncs, each restarting tests + review. That mechanism (the treadmill) was killed Aug 11 and is verifiably still off: 0 catch-up commits in 156 commits across the 40 recent changes. The freeze rule (round 2+ only pushes for blocking findings) was violated on its own motivating example the day it was written, but with advisory notes now non-blocking and few real findings per change, it's currently a minor factor.

Decisions

Multiple choice; recommendation marked. Answers save for everyone viewing this page.

1. The CI fan-out (the main remaining slowdown)

Each change now runs ~15 jobs instead of ~6, and the expensive full-suite path fires 2.4× as often. Getting green doubled from 4 → 8 min, and that cost repeats on every push of every change.

Audit the escalation trigger + collapse the 6 guard jobs back into 1–2Recommended
Find out why full-suite fires on 30% of runs (target: back near 12%), and un-split the guard checks that added 6 queued jobs per change. Keeps every gate, removes the fan-out. ~1 session of work. Make the new gates conditional
Guard pins / DB-safety gate only run when the diff touches what they guard. Bigger win, more logic to hold correct. Accept 8 min as the price of the gates
Do nothing; the gates were each added for a reason.

2. The rounds gauge

The "review rounds per PR" metric counts raw review events, so duplicate bot posts read as ballooning reviews. It misled this very investigation until we read the ground truth.

Fix the metric (count verdicts per push, not events) AND stop the double-postingRecommended
Two small changes: the pipeline-metrics script dedupes to real rounds; the review workflow stops posting 2–3 events per push. The gauge becomes trustworthy and the noise disappears from PR timelines. Fix the metric only
Cheapest; the double-posting stays but stops polluting the number. Leave as is
Treat the metric as an event counter and remember the caveat.

3. The two fixes built Aug 11 but never switched on

From the previous session: the auto-restart for checks that hang forever (~500 manual pokes in 3 weeks), and the sweeper that picks up approved-but-idle changes / expires stale holds. Both were built and left disarmed for your approval; their status hasn't changed since.

Arm bothRecommended
They target the "over 3 hours" tail, still 2× its July share. Arm with the same guardrails they shipped with. Arm only the stuck-check restarter
The lower-risk half; hold the sweeper. Keep both off
Revisit after the CI fan-out fix lands.

4. Known safety gaps from the Aug 11 incident (still open)

During that night's shipping queue: a "hold" label didn't actually stop the auto-merge sweeper, and a change can merge while a newer test run on itself is still in flight — 5 fixes merged onto a red main during a ~40-min window. Both were logged as known gaps, neither fixed.

Fix both nowRecommended
They're merge-safety holes, not speed work; small, well-scoped, already diagnosed. Card them, next week
Accept the risk a few more days.

Not diagnosed here (flagged honestly)

Merge volume fell ~70/day → ~10–30/day. Deliberate (fewer, bigger changes) or a symptom? Unknown.
Why the full-suite escalation fires 2.4× more often despite the Aug 9 change meant to reduce it — needs a read of the trigger logic, not inferred here.
The deploy leg (merged → live) runs ~18–20 min median, stable, but well above the "~2–3 min Vercel build" the docs still claim; the promote pipeline serializes org-wide. Not a regression, but the docs' number is wrong.
Freeze rule vs. reviewer guidance contradiction ("don't be discouraged when round 2 finds more") — flagged Aug 11, still unreconciled; currently low-impact.

Sources: the locked pipeline-metrics script (30-day run, Aug 18); week-by-week raw open→merge over 2,473 merged changes; job-level timing on 80 sampled CI runs (July vs Aug); round-by-round timelines of the 10 slowest changes of Aug 11–18 plus review-state classification of the 40 most recent; session transcripts of the Aug 9–11 investigations. All quantitative claims trace to those pulls; items above marked unknown/unverified are exactly that.