Where red-main time goes
Every red on main's full test, Oct 1–7 2026: how long, why, what would cut it. Measured Oct 7 from heavy-lane runs, red issues and fix PRs.
The week was 72% green, not 50%. The dashboard's 50% and 3.9 h are Oct 6 alone. Over Oct 1–7 main was red 40.8 h in 19 episodes (about 47 h a week). Median time to green was 1.9 h against a 60-min target.
No single phase owns the time. Figuring out what broke and opening the fix took 32%. Waiting for the next full run after the fix landed took 29%. Getting the fix reviewed and through the merge line took 21%. The run that spots the red took 17%.
Half the red time was several causes at once: three or more PRs' tests going stale together, or a revert stacked on a date bomb. A clean one-PR revert would have fixed 29%. Date bombs (mostly Columbus Day, Oct 12) were in 35%.
The three changes worth making
- Revert first, fix forward second: ~7 h a week. The automatic revert already exists. In 4 of 5 one-culprit reds it was held, closed, raced or never opened; only #16406 landed clean.
- A red-main fix goes straight to the front of the line: ~6 h a week. Fix PRs took 27 min median from open to merged, but 2.6 h on Oct 6.
- Freeze the clock in the 25 known test files before Oct 12: ~4–8 h a week near holidays. The new clock rule (#17276) stops new date bombs. It does not fix the 25 files already listed, and Columbus Day has already gone off 3 times.
Upper bounds that overlap: together roughly 47 → 25–30 h a week red; 95% green allows 8.
Every episode (19)
Red runs from the start of the first failing heavy run to the end of the next green one, using every heavy run (cron, CI-completed and manual). Times are UTC. Phases: detect = failing run start → red verdict or issue; notice+diagnose = verdict → the PR that finally cured it was opened; review+queue = opened → merged; re-test = merged → green verdict.
| # | Red | Green | Hours | Detect | Notice + diagnose | Review + queue | Re-test | Cause → fix |
|---|---|---|---|---|---|---|---|---|
| 1 | Oct 1 05:55 | 08:17 | 2.4 | 24 | 73 | 16 | 29 | Wave: stale tests after #14337, #14508, #14567 (left from Sep 30) → #14697 |
| 2 | Oct 1 09:48 | 12:29 | 2.7 | 28 | 51 | 25 | 57 | Wave: #14733, #14741, #14758 → #14768, #14781 |
| 3 | Oct 1 12:29 | 14:00 | 1.5 | 26 | 3 | 26 | 36 | One PR: #14754 → fix #14809 (revert #14816 held, closed) |
| 4 | Oct 1 15:45 | 20:18 | 4.5 | 35 | 111 | 31 | 96 | One PR plus a race: revert of #14856 (#14906) and a fix for it (#14933) both landed at 17:23 and broke main again → #14966. The 2-hour revert of #14887 would not apply. |
| 5 | Oct 1 20:45 | Oct 2 00:14 | 3.5 | 52 | 106 | 13 | 38 | Wave: 11 culprit issues, mostly guard and pin counts moved by different PRs → 9 fix PRs |
| 6 | Oct 2 02:32 | 05:00 | 2.5 | 22 | 59 | 8 | 59 | Wave: #14019, #14929 → #15249, #15320 |
| 7 | Oct 2 05:00 | 08:57 | 4.0 | 25 | 98 | 77 | 37 | Stacked: auto-revert of #15318 (#15353, 39 min) + Columbus Day date bomb (#15386) + two tests the revert itself broke (#15400, #15404) |
| 8 | Oct 2 12:34 | 13:21 | 0.8 | 23 | 0 | 0 | 24 | Flake: one red run, next run green |
| 9 | Oct 3 04:48 | 05:45 | 0.9 | 15 | 0 | 0 | 42 | Flaky: #15779 was blamed, but main went green before its revert landed |
| 10 | Oct 3 06:37 | 07:44 | 1.1 | 15 | 0 | 10 | 42 | Date bomb: a fixed "2026-10-02" expired at midnight → #15938 |
| 11 | Oct 3 23:28 | Oct 4 02:00 | 2.5 | 16 | 20 | 46 | 70 | One PR: #16196 non-ASCII → fix #16248. The heavy lane then ran only about hourly. |
| 12 | Oct 4 10:34 | 12:11 | 1.6 | 16 | 33 | 25 | 23 | Date bomb: test data dated Sep 20 aged out of a 14-day window → #16394 |
| 13 | Oct 4 22:12 | 23:32 | 1.3 | 15 | 10 | 30 | 25 | One PR: #16333 → auto-revert #16406. The clean case. |
| 14 | Oct 6 04:00 | 09:35 | 5.6 | 3 | 146 | 159 | 27 | Stacked: #16937 (revert #17007 closed after 4 min for fix #17022, which took 2 h in line) + #16983 (revert #17049 closed) + Columbus Day date bomb (#17046, 2.6 h open → merged) + a 64 KiB selection limit |
| 15 | Oct 6 10:46 | 12:37 | 1.9 | 19 | 36 | 28 | 28 | One PR: #16955 → revert #17069 opened at once, closed for fix #17071 |
| 16–18 | Oct 6 14:46 | 16:45 | 2.0 | 19 | 0 | 0 | 61 | Flaky test (ci-full-set-known-red): the same commit went red, green, red. #17093 was blamed and reverted (#17119), but green came first. |
| 19 | Oct 6 23:50 | Oct 7 01:54 | 2.1 | 34 | 34 | 28 | 28 | Columbus Day date bomb (#17288) plus #17001 and a stale golden file. Claimed at 00:51 (issue 00:24), fixed by #17293. Fede's side also built #17299 for the same fix. |
| Total | 40.8 | 7.1 h | 13.0 h | 8.7 h | 12.0 h | |||
What a revert could and could not have fixed
| Kind | Episodes | Red hours | Share |
|---|---|---|---|
| One culprit PR (a revert fixes it) | 3, 4, 11, 13, 15 | 11.7 | 29% |
| Wave: 2–11 PRs' tests stale together | 1, 2, 5, 6 | 11.0 | 27% |
| Stacked: culprit + date bomb + knock-ons | 7, 14 | 9.5 | 23% |
| Date bomb only | 10, 12, 19 | 4.8 | 12% |
| Flaky test or runner | 8, 9, 16–18 | 3.7 | 9% |
Even a perfect revert has a floor of about 55 min: about 20 min for the run that spots it, about 10 min to merge, and about 25 min for the next full run. The 5 one-culprit episodes took 11.7 h. At the floor they would take about 5 h, which saves roughly 7 h a week. The waves could in principle take "revert every named culprit" (#16314 does this since Oct 4), but 4 of the 4 waves were on Oct 1–2, before that existed, while the new lane was clearing debt from Sep 30.
Date bombs were in 14.4 h (35%) counting the two stacked episodes where one was on the critical path. Three of them were the same holiday: tests that pick "+6 days" or a reschedule date landed on Columbus Day (Oct 12). That will keep happening until Oct 12 passes or those files are frozen. Veterans Day (Nov 11) is next.
How each lever was sized
- Revert first (~7 h/wk). In episodes 3, 4, 11, 14 and 15 a culprit was named but the revert was held, closed for a fix, raced, or never opened (11: the fix landed inside the 2-hour window). Each one is brought down to the ~1 h floor; for episode 14 only the culprit part counts, because the date bomb still needed a fix. That saves about 6 h in 6 days. The rule: when the lane names a culprit, the revert merges, and the fix comes back as a fresh PR on a green main. Leaving fix forward as the default with a 2-hour fallback is what lost the time.
- Fix PR jumps the line (~6 h/wk). Total time beyond 15 min, summed across the 14 cure PRs, was 5.4 h in 6 days, and 2.4 h of that was episode 14 alone. This is already allowed under the CLAIM rule (ADR-0015). It is not happening reliably: #17022 and #17046 waited 2–2.6 h. CICD's audit found 37 of 44 merge-line kick-outs were false alarms, which is likely part of why.
- Clock freeze (~4–8 h/wk near holidays, less otherwise). Pure date-bomb episodes were 4.8 h, plus about 2 h on the critical path of 7 and 14. #17276 (merged Oct 7 01:16) stops new unpinned clocks. The 25 existing files are scheduled "before 11-10". Moving that ahead of Oct 12 is the cheap win.
- Smaller levers. Run the full test as soon as a fix lands, instead of waiting for the next slot: the median re-test was 36 min against a ~20 min run time, so about 4 h a week (#15523 tried this and was closed). Paging the claim through Smith: pickup took 27 min on Oct 7, so maybe 3 h a week. But #15264 removed the Slack red-main posts on purpose on Oct 2, so that is a decision for Fede, not a quick add.
Measurement notes
- The dashboard's daily series is 45.5 / 70 / 89.2 / 82.6 / 100 / 50.1% green (Oct 1–6), so the "50%" is the last day. The "Fix main" figure is that day's median, 233 min.
- The dashboard counts only scheduled runs (
ci-factory-flow,event=schedule). Since Oct 4 most heavy runs start when a CI run finishes, so the scheduled-only view merges short episodes. Oct 6 14:46 reads as one 3.9 h spell; with every run it is three spells totalling 2.0 h. - Phantom reds. 17 heavy-lane-red issues came from cancelled runs and 2 from runs that passed, mostly "no merge to blame" and closed within about 17 min. They are not in the red hours above, but each one can hold deploys while it is open. Worth a look as a triage bug.
- Sep 30 is before the heavy lane existed (first run Oct 1 05:55). Push CI on main was red for about 17 of 24 h that day across 27 spells, with at least 15 "cure red main" commits. That gauge also fails for reasons other than tests, so it is not comparable with the rows above.
- Still red? At 07:37 UTC Oct 7 main was green. #17424 (another stale golden file) was open.
Who owns what (CICD vs Gera's side)
- ADR-0015: red main has one owner, the CLAIM. Restated in the engineering bridge thread on Oct 6: "Red main keeps the single-driver CLAIM rule." It still produced a duplicate on Oct 7 (#17293 and #17299, both editing
leasing-tools.test.ts). - Heavy lane + auto-revert (#14790, #14993, #15519, #16314, #16524): built by both sides. The issues are opened by the lane, and Fede's account comments on them.
- Bridge claims on Gera's side: the release re-check cut (#17236, merged), the flaky staff-desk reschedule test, the import-cycle check, and freezing the clock in the 25 files (part 2 of the flaky-test lane). CICD: spot-reclaim kick-outs (#17231) and the kicked-out-PR watcher (#17227).
- Open: #17378 (an urgent fast-forward holds when a commit riding along has a red heavy run).