How to grade the CI/CD pipeline with five measures and real targets instead of one fuzzy score, where we stand today, and a three-part build to see the trend.
Research · measured 3 Oct 2026 · proposal, nothing built yet · review date 17 Oct 2026
Right now we cannot answer that. The CI factory pages show what is happening this minute, and the flow report shows one rolling 24 hours. Nothing keeps yesterday, so there is no trend to read. This page sets out what to measure, what "good" means for each measure, and the three small pieces of work that would let us answer the question by 17 October.
Averaging the steps feels natural, but no serious CI or delivery framework does it. DORA (Google's long-running delivery research) tracks five measures and tells teams not to combine them or set them as goals. An average lets one excellent number hide a failing one. If the queue is fast and main is red a third of the day, the average reads "fine".
The standard answer comes from Google's reliability practice. Each measure gets its own target, set below perfect on purpose. The grade is then how many targets we met, and for each one, how much of its allowance we have left. This makes "100" concrete: every target met. When we hold every target for two weeks, we raise the targets toward the published best-in-class numbers. That is the recalibration you asked about. "90 out of 100" stops being a fuzzy feeling and becomes "4 of 5 targets met, and here is the one that missed."
To tell a real improvement from a lucky day, we use a control chart. Plot each day in order and draw limits from our own normal day-to-day swing. A day only counts as a real change if it lands outside those limits or starts a sustained run on one side. That answers "is this fixed for good or just for now".
Targets are first-cut. Where a published benchmark exists it is named. Where none exists, the target starts from our own numbers.
| Measure | Target | Today | Status |
|---|---|---|---|
| Main is greenShare of time main's full test run passes | ≥ 95%CircleCI recommends 90% run success; best teams keep main green almost always | 70%red 19 of 63 hours, 1–3 Oct | miss |
| Time to fix mainRed run to next green run, median | < 60 minDORA's top tier for recovery | 1.9 h10 red spells; the 2 on 3 Oct were 40–50 min | improving |
| Reds handled cleanlyWasted queue runs caused by each bad PR before it was kicked out, median | ≤ 2 runsour own target; no public benchmark | 7 runsworst single PR: 104 runs | miss |
| Queue waitTime a PR spends in the merge queue | median ≤ 30 min p90 ≤ 60 minmerge-queue vendor guidance | 82 min 147 min | miss |
| Merged first tryPRs that merged without being kicked out of the queue | ≥ 85%our own target | 65% | miss |
Grade today: 0 of 5 targets met. That sounds bad, and it is meant to be readable. It names exactly what to fix, and it will move.
The direction is clearly good. But a control chart needs roughly 15–20 days before its limits mean anything. Three points cannot separate "the fixes worked" from "a quiet Friday". There is a second gap. Every red spell ended on newer code, but about 17 PRs merge each hour, so new code always lands between two runs. That does not prove a fix closed the spell. Telling a real fix from a flake needs the failing test's name compared run to run. The queue's health check already does this for the queue. Main does not have it yet.
A raw count of red queue runs is misleading. In the last 24 hours, 1,102 queue runs went through. Only 71 failed because of the PR being tested and 44 failed on infrastructure. Another 479 were red only because a bad PR ahead of them dragged them down. The flow report already sorts runs into these groups, so the reds chart should use them:
For the past-hour view: a small chart of the last 48 hours, one mark per real red, coloured by whether it was handled. The 14-day trend chart sits beside it.
You asked whether 90 is really the top or whether a bottleneck caps it. The data already says where the bottleneck is. 38% of all queue machine time (23,439 of 62,167 job-minutes in 24 hours) goes to knock-on runs that a bad PR ahead of them ruined. That waste is also why queue wait is 82 minutes instead of 30. Cutting how far one bad PR spreads would move three of the five measures at once. An averaged score would hide this. Separate targets make it the headline.
Almost everything is already computed and simply thrown away. So the work is mostly about keeping it, not new measuring, and it adds no new load on GitHub.
Without this, nothing else can work. Once the measures are stored over time, the trend shows up on its own.
This sits under the merge-queue flow strip that is already on the page. Nothing else on the page changes.
This also needs the fix-or-flake check for main described above.
Part 1 changes how the collector and the docs page relate (a new history store), so it needs a short decision record when it is built, under the fleet's ADR rule. Targets are reviewed on 17 October. Any measure that has held its target is tightened, and any target that turns out unreachable for a real reason is reset with that reason written down.
Not yet. What is missing today is the history itself, not a place to show it. A clickable React app built now would be a nicer view of the same single day. The CI factory pages already sit behind the company login, which sends anyone signed out to the login screen, so only the team can see them. The scorecard and reds chart fit there without a new app to host, secure and keep running.
Part 1 is the same work either way. Any future app would read the same stored history. So the cheap order is to build Part 1, put Parts 2 and 3 on the existing page, and decide on 17 October with two weeks of real data. A separate app earns its place if, by then, we keep wanting to click into one red and see its PR, its failing test, and what it cost. That kind of browsing outgrows a one-screen dashboard.
Sources: DORA metrics guide (dora.dev) and 2024–25 State of DevOps summaries; CircleCI State of Software Delivery 2025/2026; Uber SubmitQueue (EuroSys 2019); Google SRE book, "Embracing Risk"; Donald Wheeler on process behaviour charts; Mergify merge-queue guidance. DORA tier figures come from secondary summaries and were not checked against the original reports. Our numbers come from propflowai's merge-queue run cache, the 24-hour flow baseline ending 3 Oct 19:59 UTC, and the post-merge full-test runs of 1–3 Oct.