0045 — An urgent fix ships at once, unless a commit riding along with it has a red heavy run

• Status: Accepted 2026-10-07. Built in propflowai #17378 and merged on Gera's explicit go ("merge it"); merged 2026-10-07T17:50:12Z as 565f31b11. That go overrode Trinity's condition that the PR wait for Fede's answer. Option 2 was agreed by Gera's CI session and Trinity on the engineering bridge thread (C0C10AS4U15, ts 1791313627.090569, replies 1791341128 → 1791342259). Fede had not answered the options question in #agent-trinity (ts 1791344070) when it merged. If he prefers option 1 or 3, the hold is one small revert of #17378 and this ADR is amended to record his choice: a dated "Amended" entry under this Status line, linking the revert PR.
• Date: 2026-10-07
• Deciders: Gera (authorised), Trinity (verdict on options A/B/C), Fede (open for objection).
• Systems: propflowai deploy-web.yml urgent-advance, scripts/ci/release-advance.mjs, the post-merge heavy lane (post-merge-heavy.yml), and through release: Deploy Web, Deploy Lambdas and the renewal worker.

The problem

release is what production ships, and the heavy lane fast-forwards it only to a commit it proved green. The hotfix path skips that wait: after a green push:main CI run whose subject is [urgent] or Revert ", urgent-advance fast-forwards release to that commit. A fast-forward carries every commit between release and the urgent one.

2026-10-07 01:20Z: release had been parked at 42eb581f9 for about 2.5 h while main's heavy runs were red. #17285 [urgent] went green on push:main, and urgent-advance moved release to 9d0f47837 (Deploy Web run 37556553287). That carried 34 non-urgent commits to the web and the Lambdas. Four of them, 2162883bd, 17b92b2a0, ddaafb881 and 21f991354, had red heavy runs, and the heavy run on 9d0f47837 itself went red minutes later. Nothing broke in production, because all the failures were test-only. That was luck, not design. release-advance.mjs checked only that the move was a fast-forward.

Options considered

• A. Cherry-pick the urgent commit alone onto release. Ruled out by Trinity. It puts a commit on release that is not on main, and four fail-closed guards then stop the release line: planAdvance returns diverged and never forces, decide-lambda-ship.sh refuses a candidate that is not on main, the renewal worker gate refuses the same, and promote-now's dispatch guard refuses it too. One hotfix would ship, then nothing else until someone force-reset release by hand.
• B. The urgent fix waits for a heavy pass on the whole range. That removes the hotfix path. Heavy runs take 18–21 min, and last night the heavy lane was red for 2.5 h.
• 1. Keep the fast-forward as is. This is the documented design (runbook "Hotfix path", plan #15783). It is Fede's call whether to keep it.
• 2 (this decision). Fast-forward at once, unless a commit in release..urgent (the urgent commit included) has a red post-merge heavy run. In that case, hold.

The decision

1. Before the CI-triggered urgent fast-forward, release-advance.mjs urgent-check --sha <urgent> decides. It reads compare/release...<urgent> (the commits that would ride along) and the failed post-merge heavy runs (by event, status=failure). It holds (exit 10) when any commit in the range has a failed heavy run. The pure decision is decideUrgentRange, tested in scripts/__tests__/release-advance.test.ts. Pinned as a test (decideUrgentRange › "replay of 10-07 01:20Z" in scripts/__tests__/release-advance.test.ts, synthetic SHAs); replayed with live data on last night's range (42eb581f9…9d0f47837), it holds: red heavy run on 2162883bd, 17b92b2a0, ddaafb881, 21f991354, 9d0f47837.
2. On hold: release is left alone, the step prints a ::warning, and a plain message goes to #alerts. The message names the commit and the override command. The post is fail-soft, like the deploy-failed alert. The job stays green, because a hold is a decision, not a failure. The urgent commit then ships in one of two ways: the heavy lane records green on a later main tip and fast-forwards release as usual, or a human runs promote-now.
3. Commits with no heavy run yet ride along. Superseded, paced and still-running runs all count as "no run yet". Those commits passed a merge-queue train. Last night most of the 34 were in this state, not red. Requiring a heavy pass on them would turn option 2 into option 3, i.e. B.
4. promote-now (workflow_dispatch with sha) skips the check. It is the human override, and the reason for skipping is logged by promote-now itself.
5. Fail closed. An unreadable range or run list holds, and so does a range longer than the 250 commits the compare API lists. A check that crashed (any non-zero exit) holds too. A wrong hold costs one promote-now; a wrong advance ships unproven code to prod.
6. Revert " commits are held the same way. This is a choice. The cure for a red range is the revert landing and the heavy lane going green on it, which moves release normally. That costs one heavy run: 4–12 min when the train proved the tree, 18–21 min otherwise (measured 10-07). During a production incident, promote-now ships the revert at once. The #alerts message prints the exact command. This consequence is stated in #17378's description and in the merge close-out on the bridge thread, so Fede sees it when he answers.

Why a second reading of "red"

main-red-hold.sh defines main as red by one signal (#14893): an open heavy-lane-red culprit issue, and "no merge to blame" does not count. That signal decides whether pushes to main re-run and deploy. This ADR reads a different thing: whether this specific range contains a commit the heavy lane saw fail. The two disagree in exactly last night's case. #17288 was "no merge to blame", so the culprit count was 0 and the deploy hold did not apply, while four commits in the urgent range had red heavy runs. Shipping a specific set of commits to prod should be judged on those commits' own evidence, not on whether triage could blame one merge. The culprit-issue definition is unchanged for everything it already governs.

What is not changed

• The heavy lane's own advance, the proven-head skip (#17236) and the deploy hold.
• [lambda-hotfix]: it ships one commit directly to the Lambdas and does not move release. Out of scope.
• The renewal worker's direct self-ship of an [urgent] commit (it ships that commit, not a range). Out of scope.
• Branch protection. Not touched.

Stated gaps

• A heavy run that is still in progress when the urgent CI run completes is "no run yet", so its red, if it comes, arrives after the advance. Last night that was the urgent commit's own heavy run; the four earlier reds would still have held. Who hears about that red: the heavy lane's own red path, unchanged. It opens a heavy-lane-red issue, comments on the culprit PR, opens the auto-revert, and holds main's deploys while the culprit issue is open.
• A failed heavy run is counted against the commit it ran on (head_sha, which is github.sha in the resolve job). An infrastructure failure therefore holds too. That is fail-closed, and promote-now overrides it.
• The run list reads the 100 newest failures per event. A failure older than that on a commit still ahead of release would be missed. That means release has been parked behind more than 100 failed runs, which the heavy lane's own triage would surface long before.

Rollback

The change is one guard clause in one step plus one script mode. Reverting the PR restores the plain fast-forward. There is no state to clean up, no ref is rewritten, and no branch protection is involved.

Relation to other ADRs

• ADR-0003: the urgent advance reads the heavy lane's run conclusions, a signal; it never calls into the heavy lane.
• ADR-0004: this changes how Deploy Web's urgent path and the heavy lane relate, so it is written at decision time.
• ADR-0009: an unreadable signal holds; it is never read as "no red".

Build