0079 — A red sweep reverts its culprit and release keeps shipping; a red with no culprit is never reverted
- Status: Accepted 2026-11-01. Recorded at decision time (ADR-0004). Part 1 (release stops freezing once the culprit is reverted) and part 2 (the bot's revert jumps the merge queue) are propflowai PRs, listed under Build.
- Date: 2026-11-01
- Deciders: Gera, 2026-11-01, to the
cicdsession: the red-main handling is not working; build automatic undo so CI can test less before merge. Written by the worker in that session. The four needs come from the session's measurement of the last 14 days (Why), and Fede's 2026-10-29 CI plan, where "Revert-not-freeze on a red sweep" and "queue jump" were deferred out of the 48-hour rollout (docs.propflowai.co/a/ci-two-hours-2026-10-29). - Amends: ADR-0015 Amendment 2026-10-07 (what "clear" means, and that the revert is the first fix) and rule 5 (who may jump the queue). ADR-0045 decision 6 (a revert waits for a green heavy run). Nothing else in either ADR changes.
- Systems: propflowai
post-merge-heavy.yml(the sweep,find-culprit.ts,heavy-lane-triage.ts,heavy-lane-auto-revert.ts),release-advance.yml(release-promote-gate.mjs),ci.ymlpreflight (main-red-hold.sh), the merge queue.main-statusand the bridge CLAIM are unchanged.
The problem
Since 2026-10-29 the full unit suite and guard pins run as a sweep on main, once per 30-minute merge batch. A red sweep holds every release advance until a new green sweep (release-promote-gate.mjs, condition 3). The revert of the bad merge is a fix, but it too waits: the revert PR goes through the queue like any PR, then push:main CI, then a fresh 18–21 minute sweep. That is the freeze.
The CI plan (explore-test-once) wants the queue to test less, which means more breaks reach main and are caught by the sweep. That is safe only if a red sweep costs a revert, not a freeze, and only if a red with nobody to blame does not turn into reverts of innocent merges.
Why — measured, 2026-10-18 → 2026-11-01
- 7 red sweeps in 14 days. 6 named no merge (flake, environmental, unresolved). 1 named a culprit: #22502.
- #22502, 2026-10-26, UTC. The PR merged at 18:08 (main
d4454ba52, docs-only: aCLAUDE.mdline that a drift test reads).releasemoved tod4454ba52at 18:17, on its green train and green CI (docs). That was before any sweep had tested it. The 18:32 sweep went red, and its bisection named #22502 with one failing test. The heavy lane openedrevert/auto-22502(#22529) at 18:46:23, nine seconds before the culprit issue #22530. The revert merged at 19:01:42.releasedid not move again until 19:23, after the dispatched 19:12 sweep came back green. - A correction to the brief this ADR was written from: that brief said a human reverted #22502. The record says the bot did it. The branch is
revert/auto-22502, the title is the bot'srevertTitle, the author is the automation PAT's account, and the PR opened nine seconds before the issue, the bot's order (ADR-0015 Amendment 2026-10-01, Limits). The revert path worked: 14 minutes from red to revert PR. The waste was the 22 minutes after the revert merged, when the fix was on main andreleasestill waited for a green sweep. - Shipping the prefix (
release→ the culprit's parent) would have done nothing here.releasemoves on train and push CI, ahead of the sweep, so it already held the culprit at 18:17.releaseonly moves forward, so it cannot step back toculprit^. A prefix ship helps only whenreleaseis behind the culprit. That is the uncommon order now, so it is not built (Rejected).
The decision
- A culprit is clear when the bisection names it.
find-culprit.tsnames a merge only after three checks. Each failing test is re-run at the red SHA, and one that passes is a flake. It is run at the last green SHA, and one that fails there is environmental. The rest are bisected over first-parent history to the first merge where they fail. That merge is the culprit, and the sweep also retries failed files once before any of this. propflowai's auto-revert already reverts each culprit with that verdict, not only a one-merge range. This decision records that as the rule. The one-merge range of Amendment 2026-10-07 still decides one thing: whether the infrastructure gates (a cancelled shard, a missing results file) may stand the revert down. - What reverts: the culprit's squash merge, by the heavy lane, on the run that names it, as
revert/auto-<pr>, auto-merge armed, culprit PR commented. These rules are unchanged and still refuse a revert: never revert a revert; never revert one already reverted on main; never re-open a revert a human closed; a red made only of bookkeeping guards (a count or ratchet) gets an issue, no revert; at most 5 bot reverts in 24 h. - No merge to blame (6 of 7) is never answered by a revert. Flake, environmental and unresolved verdicts revert nothing, ever. The path is: the sweep's retry-once; the
Heavy lane red: no merge to blameissue; and the red-main owner fixes the test or deletes it (Gera's flaky-test rule: a real root-cause fix or a delete, never skip, quarantine or retry-to-green). A date bomb is fixed forward (Amendment 2026-10-07 decision 4).releasestays held until a green sweep, as today. The freeze is still correct there: nothing has been fixed. - Revert-not-freeze. A red sweep is ANSWERED once every culprit its own bisection named is reverted on main, and an answered red stops holding
release. The gate reads the red sweep'sheavy-culprit-reportartifact. It needs verdictculprit, no environmental test, and, for every culprit, a commit at or before the tip that reverts it and was not itself reverted. A revert is found by git'sThis reverts commit <sha>line, or by aRevert "<culprit title>"subject, the two markersalreadyRevertedOnMainreads. The tip must still have a green merge-queue train and green push CI, so nothing else about the gate changes. Anything missing or unreadable holds. The dispatched re-test still runs, and if that sweep is red, the new red holds on its own. - A revert commit is not held by the main-red deploy hold. The auto-revert's PR title starts
[urgent], but the squash subject isRevert "…"(#22529 landed without[urgent]). So the hold inmain-red-hold.shcancelled the revert's own push CI, and decision 4 needs that run green. ARevert "subject now counts as urgent there, the same waydeploy-web's urgent path already treats it. This narrows ADR-0045 decision 6 for the normal release path only. Itsurgent-checkon the hotfix path is unchanged. - The claim. The owner is still one owner, recorded as ADR-0015 and ADR-0074 say: a
main-statusdeclaration, and the heavy lane'sBRIDGE CLAIM v1withdriver: smith. GitHub Actions can read neither. Inside GitHub, the openheavy-lane-redissue whose title names the culprit is the readable form of "this red is claimed". That is the same move Amendment 2026-10-01 made for thefix-forwardlabel. The heavy lane acts for the owner on one thing only: the revert of a culprit it named. - The bot's revert jumps the merge queue while that culprit's issue is open. The revert PR gets the
auto-revertlabel when it opens. Once it can be queued (review verdict and PR checks green), it is enqueued withjump: true. If it is already in the queue, it is dequeued and re-enqueued with the jump. Four things refuse the jump, and the PR then waits its turn like any other: the culprit'sheavy-lane-redissue is closed; the head branch is notrevert/auto-<pr>; the PR's commit is already on main (re-queueing would merge an empty second commit); or the PR is not open. The jump cannot happen when the revert opens, becauseenqueuePullRequestrefuses a PR that is not mergeable yet. - The standing rule is unchanged for everyone else: no
jump: truewithout holding the red-main claim. Decision 7 is the claim holder's own action, taken by the lane that holds the GitHub-side form of it. It is not a door for sessions or humans. A session that wants a jump still declares the red first.
Why a jump is worth its cost
A jump restarts every train already in line. On 2026-10-06, #17178's jump rebuilt 8 trains. While main is red, every one of those trains carries the culprit, so a train that merges first does not change main's state, and the revert has to land before anything is fixed. Restarting trains behind a red main costs runner minutes. Leaving the revert at the back of a 20-PR queue costs release the whole queue. The jump is used once per culprit, only while the culprit's issue is open, and only for the bot's own revert.
On #22502 the jump would have saved little: the queue was quiet, and the revert's train ran in 3 minutes (18:58 → 19:01). Its 12-minute wait before that was the review and PR checks, which the jump does not touch. The jump matters on a busy afternoon, which this incident was not.
Rejected
- Ship the prefix (
release→culprit^). It is a no-op whenreleasealready holds the culprit, which is the normal order sincereleasemoves on train and push CI (Why). It would add a second release path to keep correct, for the uncommon case wherereleasewas behind. - Revert on any red sweep. 6 of 7 reds named no merge. Reverting on those would revert innocent merges, because the blame range is a half-hour batch of 3–8 merges.
- Wait the 2-hour fallback for a multi-merge range. The bisection already re-ran the failing tests at both ends. Waiting two hours buys nothing deterministic that the bisection did not already prove.
Stated gaps
- Tests that passed on the red sweep are not re-proven at the tip. An answered red lets
releaseship a tip whose own sweep has not finished. That is the same trust the gate already gives a tip after a green sweep: train green, push CI green, sweep at most 30 minutes behind. The dispatched re-test covers the rest, and a red re-test holds again. - The artifact can expire or be missing (14-day retention, a crashed report step). Then the red is not answered and holds, the old behaviour.
- A revert written by hand with neither marker (no
This reverts commitline, noRevert "<title>"subject) is not recognized. It holds until a green sweep, as before. - The jump reads the issue, not
main-status. A human who declared green while the issue is still open does not stop a jump. The issue closes itself on the next green sweep.
Build
| Part | Repo | What | State |
|---|---|---|---|
| ADR | agentflow | this file; register update for the 0079 citation that was propflowai's | this PR |
| 1 | propflowai | release-promote-gate.mjs redSweepAnswered + gather reads the red sweep's report; main-red-hold.sh lets a revert's push CI run; tests incl. the #22502 replay and its controls |
PR opened with this ADR (see the PR list on ci/release-answered-red) |
| 2 | propflowai | auto-revert label + jump with the claim held (decision 7) |
PR opened with this ADR (branch ci/auto-revert-jump) |
Relation to other ADRs
- ADR-0003: the gate reads the sweep's artifact and main's history, which are signals. It never calls the heavy lane.
- ADR-0009: an unreadable report or range is never read as "answered".
- ADR-0015: one owner, unchanged. Decisions 1, 6 and 7 narrow Amendment 2026-10-07 and rule 5, as stated above.
- ADR-0045: the hotfix path's
urgent-checkis unchanged. Decision 5 here changes the normal release path only. - ADR-0074: the declaration is still the record. Decision 6 says what GitHub reads in its place.