PropFlow.ai PropFlow.ai Docs
Unknownmarkdown source

ADR-0107: Deploy-gate relocation — retire the merge queue, gate deploys on green main

Date: 2026-07-21 Status: Accepted

Context

Since 2026-07-14 (see the cost-model history in .github/workflows/ci.yml), the GitHub merge queue was the authoritative full-suite gate: PRs ran a capped affected subset, every queue entry ran the full sharded suite on the merged tree, and push:main ran no tests. Measured reality after a week:

The obvious "just drop the merge-queue run — the branch already tested it" is unsafe here: PR CI runs the capped affected subset, and hub-module PRs (selection > AFFECTED_CAP) skip PR-level testing entirely, deferring to the queue. Deleting the queue with no replacement ships those PRs to prod with zero test coverage until nightly. And un-capping the PR run is the already-measured 2026-06-20 model that cost $1k+/mo.

Decision

Move the full-suite gate from in front of every merge to in front of deploys. Same protection for production, zero toll on merging:

  1. Merges are instant. The merge queue is removed from the main ruleset (and strict up-to-date checking stays off). A PR merges the moment its PR checks are green.
  2. push:main runs the authoritative full 12-shard suite, plus type-check, coalesced by a concurrency group keyed on (workflow, event_name, ref) with cancel-in-progress: true: a burst of N merges cancels the superseded runs and one run tests the tree containing all N — this is the batching the queue was supposed to provide, with no train bot and no hourly clock. Expected ~15–25 authoritative runs/day vs the queue's ~63 + requeues (~60–70% cut of that bucket).
  3. Deploys gate on that run's conclusion:
    • Vercel prod: scripts/vercel-ignore.sh no longer auto-builds main; the vercel-promote job in ci.yml creates the production deployment via the Vercel API only after the suite is green. (Verified empirically 2026-07-21: API-created git deployments bypass the ignoreCommand, so the skip cannot deadlock the promote.) Prod web builds also drop to one per green run instead of one per merge.
    • Lambdas + Fargate worker: the await-main-ci composite action (.github/actions/await-main-ci) blocks each deploy's gate job until the push:main run for that SHA — or, when coalescing cancelled it, the newest superseding main run — concludes, and fails loud on red. workflow_dispatch deploys skip the wait (dispatch is the human override). This flips the premise of the PR #4157 rejection: gating on push:main was "always-true theater" only because that run executed no tests; now it is the authoritative gate.
  4. A red main alerts (alert-main-red → Slack #alerts) — deploys hold automatically, a human reverts or fixes forward. Expected rarely: the only new exposure class is a hub-PR failure that the queue would have caught pre-merge, and the week's data says those were ~all flakes.
  5. The group-diff subset machinery dies with the queue — select-tests, plan-selected, test-selected, scripts/ci/select-tests*.mjs, the vercel-merge-group-stub, and the merge_group stubs in ci-nested.yml / claude-code-review.yml are deleted (one source of truth; the machinery only ever ran on merge_group events).

Cutover order (corrected by bot review on #4303): flip the ruleset BEFORE merging this PR. A merge-group run evaluates workflow files on the merge-group ref — which contains this PR's deletion of every merge_group trigger, so none of the required contexts (Build, Type Check, Unit Tests, review, Nested Gate) would ever report inside the queue; the entry would stall to check_response_timeout and get ejected. The PR cannot ride the queue it deletes. Sequence: (1) API: remove the merge_queue rule from ruleset "Require CI", drop the Vercel required context (previews are opt-in — with the queue's stub gone, a non-[preview] PR would block forever on it), keep Build / Type Check / Unit Tests / review / Nested Gate, strict: false (requiring up-to-date branches would reintroduce the re-run treadmill; the post-merge run covers stale-base drift). (2) Merge this PR directly on green PR checks. Between (1) and (2) merges are briefly un-queued while main still has the old ci.yml — its push:main run is the old no-op, and deploys in that window behave exactly as before the relocation (ungated), which is the status quo, not a regression.

Consequences

Drift-guarded by src/__tests__/ci-cost-routing.test.ts (rewritten for this model: push:main leg present, coalescing on, no merge_group trigger, promote

Revision 2026-07-23 — affected-on-main; coalescing retired

Measured reality after 2 days: coalescing never coalesced. Merges drip ~12+ min apart while the suite runs ~8–12 min wall, so runs rarely overlap — 46 full-suite runs completed on Jul 23 (65 fired, 19 cancelled), 51 on Jul 22, vs the ~15–25/day this ADR assumed. Each is ~45 job-min; the July Actions budget hit 172% by the 23rd. Same failure the merge queue had (assumed batching that arrival cadence never delivers), relocated.

New model (same safety invariants, ~10x cheaper on the common path):

Expected spend: ~35–50 full runs/day → a handful of hub-merge escalations + per-push ~2–6 min affected runs. Drift guard updated in the same PR (ci-cost-routing.test.ts: escalation leg + mode wiring + per-push groups pinned).

Deploy ordering under per-push runs (added in review)

No-cancellation means two overlapping push runs can both reach vercel-promote. Each deploys its own GITHUB_SHA, so without a guard a slower-earlier run could promote an older SHA after a newer one — rolling prod backward. vercel-promote therefore carries a tip-of-main guard: it promotes only when its SHA is still the branch tip, otherwise it skips and the newer push's run owns the deploy (the same outcome cancellation used to guarantee). A tip-lookup failure holds the promote (fail-closed). Residual TOCTOU window is the seconds between the tip check and the API call — vs minutes without the guard; full serialization would need a deploy queue, which a 2-dev merge cadence doesn't justify. The same ordering question for await-main-ci consumers (Lambda/Fargate deploys) is tracked as a follow-up.

Revision 2026-08-07 — hub-sized PRs are tested BEFORE they merge (#5518)

The Context above names the hole and then accepts it:

hub-module PRs (selection > AFFECTED_CAP) skip PR-level testing entirely, deferring to the queue.

The queue went away and the deferral moved to push:main. That is a real gate and it fires after the merge, so the failure mode it leaves is not "a PR merges untested" but "a PR merges untested and the required Unit Tests check says it passed" — the affected lane skipped itself and decide-unit-tests-conclusion.sh reported completed/success, titled "Affected selection capped".

That bill came due on 2026-08-06/07: three capped PRs broke main inside ~36 hours (#5480, #5392, #5485 — the last held every deploy for ~100 minutes), and eight open PRs sat in the same state. Worse for the author, a hub-sized PR had no way to obtain unit-test evidence at all before merging: the affected lane refuses by size, test-full could not run on a non-Dependabot PR event, and the run-tests label does not help — it starts the affected lane, which caps again.

Change: an over-cap selection on a PR's verdict event (🟢 / 📝 / 🟡 or an approval) emits mode=full and runs the 12-shard suite on the PR head; test-aggregate and the required check then read test-full, not the lane that skipped itself. This is the leg that produced the false green — an ordinary PR push never reaches the cap at all (the approval gate returns deferred before the selection is computed, and deferred was already PENDING), so the only run that emitted capped on a normal hub-sized PR was the verdict run, and it wrote success.

SUPERSEDED 2026-08-09 — this paragraph describes the approval-gate era. The affected lane now runs on every ordinary pull_request event, in parallel with the review (scripts/ci/decide-test-lane.sh), so a hub-sized PR push DOES reach the cap and mode=capped is its normal per-push outcome. The escalation is unchanged: the check stays PENDING and only the verdict event escalates to the sharded suite (or the delta lane). The false-green analysis above is history and stays as written.

capped survives on one path only: a pull_request push carrying the run-tests label. There it now reports PENDING rather than green (I8) — "nothing tested this" and "this passed" stop sharing a representation, which is invariant I1 of the chokepoint applied to the cap. That half is not cosmetic either: the label is sticky, so the green it used to leave on the head SHA would sit there for auto-merge to consume the moment a verdict armed it, minutes before the escalated suite finished. (I8/I9 in scripts/ci/decide-unit-tests-conclusion.sh.)

Cost — additive, not offset. ~60–96 billed min per capped verdict run (12 shards; an estimate from the 12 × ~5–8 min shard times this suite has run at, not a measurement of this change — re-measure once a real capped verdict has run). This diff does not remove the post-merge push:main escalation, so a hub-sized change is now billed two full-suite runs — one pre-merge, one post-merge — plus one more per additional verdict round. The approval gate bounds it to once per verdict, not once per push, which is what keeps it from becoming the un-capped 06-20 model.

"One more per verdict round" understates it slightly: PR refs keep cancel-in-progress, so a second review event on the same PR kills an in-flight 12-shard run and its partial shards are still billed. A hub-sized PR driven through several review rounds realistically pays 3–5 escalations plus partials, not one.

The counterweight is a prediction, not a saving already banked: during a red-main window every push escalates, so each hour main stays red bills full runs on unrelated merges (9 such runs on the night of 2026-08-06, when #5485 held main red ~100 min). Catching those regressions pre-merge should remove most of that. Re-measure by 2026-09-07 and record the delta here; if the escalated verdict runs have not paid for themselves, the honest response is to revisit AFFECTED_CAP or shard count, not to restore the false green.

Deliberately NOT done, so nobody re-derives them:

Trade accepted: a hub-sized PR waits for the sharded suite after its verdict before it can merge — the 12-shard suite's own history in this file puts that at ~8–12 min wall, and the job's timeout is 25 min. Before this it merged instantly and detonated on main. fail-fast: false localises a flaky shard, and re-toggling draft→ready re-fires the verdict (and so the suite).

Drift-guarded by ci-cost-routing.test.ts (the verdict leg on test-full, bound to mode=full; the ordered push → verdict → capped legs inside the over-cap block; the test-full gate in test-aggregate, keyed on mode to match the script) and by decide-unit-tests-conclusion.test.ts, which executes the real decision script — including the reversed E8 row in its exit inventory.

Exercised end-to-end before merge. The decision logic is proven by tests that execute the real script; the wiring could not be, because this change's own PR selects 3 files and can never reach the cap — it cannot fire its own new leg. That gap mattered more than it looks: had the leg silently not fired, the result would not have been a false green but a fleet-wide wedge — test-full skipped ⇒ test-aggregate exits 1 ⇒ I9's case misses ⇒ emit none ⇒ the required check keeps its PENDING on every verdict, forever. Fail-closed, correctly, and precisely the "merge-blocking state with no self-service exit" the capped branch was rewritten to end — applied to every hub-sized PR at once.

So it was observed rather than inferred, on a throwaway branch carrying this change plus AFFECTED_CAP=1 (PR #5533, closed unmerged, branch deleted):

what observed
run 31197821193, event pull_request_review, head 1f2478188
gate Approval gate: running the affected lane (event=pull_request_review, state=commented)
selection 3 test files (cap: 1) → mode=full
sharded suite Unit Tests (full 1/12) … (full 12/12) — all 12 ran and passed on a PR event
decision status=completed conclusion=success — mode=full — the sharded full suite is the gate for this run, and it passed
write Patching existing Unit Tests check run 92930435607 — patched in place, so still exactly one per SHA
final Unit Tests → success, output title "Full suite passed"

The same run corrected a claim: the pre-verdict PENDING it produced is deferred, not capped. An unlabelled pull_request push short-circuits at the approval gate before the selection is computed, so it never reaches the cap — capped is reachable only via the sticky run-tests label. (Superseded 2026-08-09: with the lane running on every ordinary PR event, an unlabelled push reaches the cap and capped is the normal PENDING for a hub-sized PR. deferred now means draft-or-bot.) I8 is still load-bearing on that path (the stale green it used to leave sits on the head SHA for auto-merge to consume) but this drill did not exercise it.