Why a PR goes red in the merge queue

Source. scripts/ci-flow/queue-vs-pr-join.ts over every merge_group run of ci.yml in the window: 2,086 runs, 759 with a red cause job, 850 red-job events = 543 inherited from an earlier group in the chain + 295 first-in-chain + 12 ambiguous. Only the first group a failure appears in names a culprit (flow-baseline's real_failure); cascades are never counted. That gives 240 first-fails (one per PR × failing check or test file). For each, the culprit PR's own pull_request runs at the head it was enqueued with were read, including the vitest logs, to answer: did the job run, did the file run, did it pass.

Rules, written before the numbers. Excluded: the same signature first-failing for 2+ PRs on one base (main was red), or an unlanded entry ahead in the chain failing the same signature (collateral). C: a runner/spot/shutdown signature in the job log or in the CI factory's per-run facts, a flaky-test issue naming the file, or the same enqueued PR head later passing that check in another group. A: the job or file did not run on the PR (affected lane capped, always-run step skipped, fence subset pruned the pin, next build deferred). B: ran and passed on the PR, or the test was added/edited on main after the PR's last validation, or a type error that only the queue's cold tsc (inside the Build sub-check) reports. D: the rest.

Where this can be wrong.

• Window. Three days, after Fede's 10-03 change (the queue stopped running the full suite on every train; it now runs the affected selection and escalates to the full set only over the cap). Before that, A was structurally larger. The 09-30 → 10-02 regime is not bucketed: the join spent 2,613 uncached API calls on this window alone, against a 1,500 budget, so no second window was run. The factory's facts for that period (1,090 red runs, not culprit-attributed) show the red jobs were Guard pins 551, Full set (one machine) 364, CI Checks 206.
• C may hide B. 10 rows are C because the same PR head passed on re-queue. After 10-03 a re-queue builds on a new base and the queue's own selection can skip the file, so a pass there does not prove the test ran. Up to 10 rows (7%) could move to B.
• B is partly soft. Of the 14 test-file B rows, 6 have main touching the test or one of its first-level imports between the PR's validated base and the queue base (confirmed). 8 do not — those are "B or an undetected flake". The 3 type-error rows are B-likely: for #15900 the PR's own Type Check (tsgo) passed the same head three hours earlier and the union it switches on was touched twice on main that morning, but a tsgo-vs-tsc tool difference is not excluded.
• Control. flow-baseline's rolling 24h names 34 culprit PRs; 33 appear here. Its one extra (#16299) the join did not attribute (not investigated); the join's 24 extras are all rows this page excludes as collateral or files as infra.
• Unit. A row is PR × failing check/file. A PR with both an A and a C row rolls up to A. Gera's "1 in 4" is the first-entry merge rate: flow-baseline reads 81% over 24h (so ~1 in 5 PRs needs a second entry); the join's 57.5% also counts first runs that were merely superseded.
• Budget. 2,613 uncached calls by the join, ~220 by a stale-base pass stopped early, 2 by hand. The GitHub rate-limit endpoint read 5,000 remaining throughout, which is not explained.

61 rows: the PR's affected selection was over the cap, so the PR ran a narrower list and the queue ran the full set (mode=full). 6 rows: the always-run drift/decide step was skipped on the PR (capped or deferred). 2 rows: the guard fence subset pruned the pin on the PR. 57 distinct files; the only repeater is scripts/escalation-crucible/__tests__/registry.test.ts (5 PRs — a registry test that any new entry breaks).

• #15837 · run 37090913045 — Full set (one machine): three leasing tests (leasing-prospects-route, load-mirror-prospect-detail, load-prospects-projected.reads) failed; the PR's affected lane was capped and ran none of them.
• #15812 · run 37086054251 — Guard pins: scripts/__tests__/guards-for-paths.test.ts; the PR ran the fence subset only and the pin was pruned.
• #15983 · run 37125475904 — Guard pins: tool-dispatch-inventory.drift.test.ts; the PR's always-run drift step was skipped (capped), the queue ran it. Same file hit #16632 two days later.

Fix. When the PR lane is capped, run the full set on the PR once — at approval (or at ready-for-review), not on every push. That is 61 of 146 culprit rows (42%) removed before the queue, at one full-set run per capped PR.

11 rows ran and passed on the PR's own CI; 3 are tests that landed on main after the PR was last validated (the PR's tree never had them — a freshness class, not a selection class); 3 are type errors the queue's cold tsc found inside the Build sub-check. Commits behind at queue time, test rows: 3, 9, 9, 18, 20, 22, 25, 25, 25, 28, 28, 131, 323, 323 (median ~25, matching the ~15 median lag of the fleet's PRs only loosely — the B rows skew stale).

• #15730 · run 37133895061 — Unit Tests (affected): two ws-lead-outcome replay tests passed on the PR (validated 10-02 21:01Z), red in the queue 10-03 15:37Z, 323 commits behind; main had touched their imports (escalation fix fe2b113a, identity merge 0d8a925a). Confirmed.
• #16709 · run 37356817885 — Guard pins: module-setup-refused.drift.test.ts passed on the PR; 20 commits behind; main touched an import (b868b2b8, maintenance approval wait). Confirmed. This PR then cascaded into 5 groups behind it.
• #15900 · run 37104118391 — Build sub-check: staff-asked-application-send.ts(108,14): TS2678 in the PR's own file. The PR's Type Check passed the same head at 03:42Z (run 37094125904); 'no-guest-card' was touched on main by #15845 at 03:24Z and #15943 at 07:07Z. Six PRs behind it went red on the same error (11 runs). Two more episodes like it: live-deps.test.ts (#16321 behind #16356, 29 runs) and tour-lifecycle-cross-surface-agreement.test.ts (#16291 after #16283 landed, 9 runs).

Fix. Re-validate on fresh main before queueing — it would have caught the 6 confirmed rows and probably the 3 type errors (49 red runs between them, plus 2 on a fourth file not traced to a culprit — the costliest rows on this page per culprit). For the 8 unconfirmed rows the honest statement is "B or an undetected flake"; a fresh-main re-run also settles that question, so it earns its place either way. The pre-queue type check being built addresses exactly the 3-row class; this page says that class is small in rows but large in cascade.

35 rows: the runner died — "AWS interrupted the EC2 Spot instance" or "The runner has received a shutdown signal" — 4 on 10-03, 16 on 10-04, 15 on 10-05, after the 10-03 move to spot everywhere. 24 of the 35 are on the CI Checks job (type check, lint, DDB escapes, build), the longest-lived single job in the queue leg. 20 rows: flaky — 10 named by a flaky-test issue, 10 where the same PR head passed on re-queue. Four files carry most of the flake: leasing-compare.test.tsx (unhandled error, 4 PRs), ci-unit-tests-writer-vacuous-green.test.ts (4 PRs), self-review-unresolvable-range.test.ts, check-availability.test.ts.

• #16232 · run 37163312836 — CI Checks: AWS interrupted the spot instance; the PR's own run had passed every check.
• #16693 · run 37342988050 — Guard pins shard lost to a spot reclaim; no test result established.
• #16457 · run 37255465469 — Guard pins: ci-unit-tests-writer-vacuous-green.test.ts, open flaky-test issue, first-failed for 4 different PRs in 3 days. (#15779 · run 37083683201 is the re-queue-passed shape.)

Fix. The spot-loss fix (retry a reclaimed job inside the run, or on-demand for the CI Checks job alone) is worth 35 of 146 culprit rows (24%) and is the right fix. The flake half is four files; quarantining them removes ~14 rows.

• #16128 · run 37167504831 — Guard pins: handoff-mode-write-fence.drift.test.ts was already red on the PR and the PR was queued anyway.
• #16733 · run 37359348670 — only the Merge-queue contexts aggregator is red; it restates other jobs, so nothing names a cause here. Same shape in #16779.
• #16785 · run 37373318524 — Guard pins red with no failure annotation and no file; unattributable.

From the CI factory's per-run facts, 09-30 22:40Z → 10-03 00:00Z: 2,926 merge_group runs (1,080 green, 1,449 red, 84 cancelled, 313 unread); red jobs across the 1,090 red runs with facts: Guard pins 551, Full set (one machine) 364, CI Checks 206, Unit Tests (affected) 9. Spot or runner-shutdown signatures in that period: 5 of the 1,090 red runs, against 63 of 774 after the 10-03 move to spot everywhere. These are red-job counts including cascades, not culprit attributions — a like-for-like A/B/C/D for that window needs another join run (~2,500 calls).