Flaky tests — the look-back

Every merge-queue train since the queue went live (30 Sep) and every pull-request run since 12 Sep 2026: which tests went red and then green on code that had not changed, and why.

2026-10-06 · 28,732 CI runs read (5,927 queue trains, 22,805 PR runs) · written by Gera's CI-quality session · companion to #17276 (the clock ratchet) and the flaky-tests reviewer lens

The one thing

Corrected 7 Oct: the 13% "load flake" bucket was a counting error; those were real breaks on 2 Oct. True load flakes: ≤2.6%.

About the correction

GitHub keeps only 10 failure annotations per check run; the dropped one was the "failed twice, not a flake" line, so retried reds were scored as passes. Job logs show 14 of 14 sampled trains failed twice. Only 6 real "passed on retry" lines exist in the whole set, all connect-caller-tool.

Most reds in the merge queue were not flaky tests; they were trains testing on top of a main that was already red. 2,932 of 5,927 trains went red; 2,842 of them were green on the next try with no new commit. 46% (1,311) of those carried the same assertion as the train before them (main, or the train's base, was already broken by a landed PR); 4% (115) were a repo-wide guard reading a committed list, sidecar or workflow a sibling PR had just changed; 13% (368) were real reds on a red base in one window on 2 Oct. A date read from today cost 7 trains in this window plus the 8 Columbus Day trains. The queue hides every one of these the same way: an ejected train shows as cancelled, not failed.

2,932

queue trains that went red (of 5,927)

2,842

of those were green on the next try with no new commit

124

distinct test files the queue named

619

test files on main that read the clock unpinned today

Where the red came from

Each red train's failed jobs were read from the check annotations and the test files they name were grouped by cause. A train can carry more than one file.

Main (or the train base) already red 1337 (46%)Committed list / sidecar / workflow drift 117 (4%)Red base, 2 Oct window (was: load flake) 386 (13%)Timers / timing 4 (0%)Clock / date (unpinned) 7 (0%)Other assertion 22 (1%)File gone or renamed since 2 (0%)

Cause	Test files	Red trains	…green unchanged	What it looks like	The fix to ask for
Main (or the train base) already red	69	1337	1311	The same assertion fails on train after train until a fix lands: a landed PR broke a test nothing ran on the PR, and every train behind it pays.	Run the repo-wide guards on the PR (#17075, landed 6 Oct); read the FAIL line and run the file on main before blaming the PR.
Committed list / sidecar / workflow drift	13	117	115	A drift or ratchet test reads a hand-kept JSON, a generated sidecar or the workflow files; a sibling PR changed the thing it pins, so every train behind it is red until the list catches up.	Derive the set at test time (#17268 shape) or generate the sidecar in CI; never a committed list several PRs rewrite.
Red base, 2 Oct window (was: load flake)	23	386	368	Corrected 10-07: job logs say "failed twice, not a flake" on 14 of 14 sampled trains. Three deterministic breaks on 2 Oct (the ci.yml test-aggregate step at 19,880 bytes, a new ownerId credential site, a broken absentMeansFrom citation), all fixed that day.	Fake timers + advanceTimersByTimeAsync; assert order, never duration; no shared state across files.
Timers / timing	2	4	4	A wall-clock bound or a fixed sleep that holds on a quiet laptop and not on a saturated box.	vi.useFakeTimers + advance; vi.waitFor on the condition; no setTimeout sleeps.
Clock / date (unpinned)	3	7	7	A fixture or expectation built from new Date() / Date.now(); the holiday, month end or DST switch arrives later.	vi.useFakeTimers({ now: FIXED }) or setSystemTime; an explicit base date in the test. Ratchet: #17276.
Other assertion	13	22	22	A real assertion failure the annotations do not explain; usually code a sibling train changed under the test.	Read the failed job log; if the test asserts on another PR's surface, it is the queue doing its job.
File gone or renamed since	1	2	2	The test no longer exists on main under that path.	—

Window caveat: the merge queue itself is only 7 days old (30 Sep), and 2 Oct alone, the day the one-machine full set landed, is half the red trains. The queue has retried a failed file once, and quarantined one that passes on retry, only since 2 Oct (#15203, #15311). Before that a flaky file was simply a red train. "Passed on retry" and "failed twice" can therefore only be counted over 4 of the 25 days.

The tests that cost the most trains

Test file	Red trains	Green unchanged	Failed twice	Passed on retry	Cause
`src/__tests__/sync-tour-mock-covers-both-views.drift.test.ts`	308	305	0	103	Main (or the train base) already red
`src/__tests__/workflow-expression-length.drift.test.ts`	210	193	0	210	Red base, 2 Oct (was: load flake)
`src/__tests__/guard-receiver-dot-blindness.drift.test.ts`	183	180	0	31	Main (or the train base) already red
`src/__tests__/legacy-field-elimination.drift.test.ts`	170	169	0	0	Main (or the train base) already red
`src/app/api/leasing/renewals/approve/__tests__/approve-scope-no-job-title-gate.test.ts`	170	169	0	0	Main (or the train base) already red
`src/lib/domain/portfolio/registry/__tests__/absent-means-provenance.drift.test.ts`	154	142	0	149	Red base, 2 Oct (was: load flake)
`src/__tests__/property-owner-id-retired.drift.test.ts`	117	109	0	69	Red base, 2 Oct (was: load flake)
`src/__tests__/identity-journey-one-person.test.ts`	113	112	0	0	Main (or the train base) already red
`src/__tests__/identity-journey-application-join.test.ts`	98	98	0	0	Main (or the train base) already red
`src/__tests__/decide-deploy-queue-stuck.test.ts`	96	93	0	0	Main (or the train base) already red
`src/__tests__/scoped-reads-note-the-grants-epoch.drift.test.ts`	62	62	0	0	Main (or the train base) already red
`src/lib/domain/portfolio/context/__tests__/raw-store-importers.drift.test.ts`	49	49	0	1	Committed list / sidecar / workflow drift
`src/__tests__/move-conversation-property.test.ts`	41	41	0	0	Main (or the train base) already red
`lambda/appfolio-sync/__tests__/handler-read-attribution.test.ts`	38	38	1	0	Main (or the train base) already red
`src/__tests__/workspace-page-streaming-fallback.drift.test.ts`	37	35	10	0	Main (or the train base) already red

What the retry-and-quarantine lane saw (2–6 Oct)

Since 2 Oct a file that fails once in the queue's full set is re-run alone; passing on the retry opens a flaky-test issue and quarantines it. That record is the only direct count of load flakes we have.

Quarantined file	Reports	Shape
`src/lib/domain/leasing/__tests__/google-maps-failure-loud.test.ts` #16276	1	no clock, no sleep: shared-state / order
`src/__tests__/check-availability.test.ts` #16242	2	fake timers in 6 blocks, 7 clock reads, no fixed instant
`src/__tests__/leasing-context-injection.test.ts` #15852	0	19 clock reads, no pin, a 60 s timeout
`src/lib/integrations/email/__tests__/org-mailbox-address-key-filing.test.ts` #15851	3	no clock, no sleep, no network in the file: shared-state / order
`src/__tests__/connect-caller-tool.test.ts` #15593	30	fake timers + Date.now() deltas asserted after advanceTimersByTimeAsync; the top load flake by a mile
`src/__tests__/initiate-renewal-call.test.ts` #15590	0	no clock, no sleep: shared-state / order
`src/__tests__/sync-renewal-voice-prompt.test.ts` #15561	1	no clock, no sleep: shared-state / order

The pull-request lane

22,805 PR runs, 4,975 failures. 399 commits had a failed run and a green run on the same commit. That number is a ceiling, not a flake count: a PR run tests the merge of the branch with that moment's main, so the same commit can fail on main's breakage and pass an hour later. The failing jobs on those runs were overwhelmingly the aggregator jobs (Unit Tests, Unit Tests (decision)), whose annotations do not name a test file; the file names live in the shard logs, which this look-back did not read.

Gera's two suspects

Time and date: real, small in count, largest in blast radius. 3 files, 7 red trains in the window, every one a "failed twice" (deterministic on the day, so the retry could not save it); plus the Columbus Day file (staff-desk-reschedule-cancel-wiring, 8 trains on 6 Oct, fixed in #17263) and held-text.test.ts, which failed twice on 3 trains with new Date(Date.now() ± …) fixtures. Latent population on main: 619 test files, 1,773 unpinned clock reads; 42 install fake timers at the real instant, which is still today. Of these, 115 compute a date relative to today (the exact Columbus Day shape). The ratchet in #17276 stops the population growing; each file shrinks it when touched.

Exact wording on model output: not a CI flake source. 0 red trains in the window. The 10 tests that call a real model are gated behind RUN_EVALS and never reach the unit lanes; the 33 files that assert exact text near a model call do so on mocked replies. No ratchet is warranted; the reviewer lens asks for decision-level assertions instead.

What changed because of this

#17075 (already landed) — the derived always-run set: every repo-wide guard runs on the PR, so the "main already red" bucket is the one that should shrink first from here.
#17276 — the unpinned-clock ratchet: a test file a PR adds or edits may not have more unpinned clock reads than its base version had. No committed baseline (the committed-list class above is exactly why).
Reviewer lens flaky-tests.md — blocking rules for the five patterns, each with the fix to ask for, loaded on every PR that touches a test file.
#17263 — the Columbus Day file pinned at a fixed instant, with a control replaying the ejection's exact time.
#17268 (in flight) — the import-cycle baseline derived at test time instead of committed; the same medicine for the biggest bucket here. The other committed lists in the top table are candidates for the same treatment.

Method

Runs listed from the Actions API in six-hour windows (the API caps a single listing at 1,000). A red train = a merge_group run that ended failure or cancelled with at least one failed job; the 144 cancelled trains with no failed job were collateral re-queues behind an ejection and are not counted. "Green unchanged" = the same PR's next queue run succeeded and the PR received no push in between. Test files come from the failed jobs' check annotations (failed twice, not a flake: …, retrying once: …, assertion messages). Causes were assigned from the annotation text first, then from the test file's source on main. Latent counts come from scripts/lib/unpinned-clock.ts --census (syntax-based, comments and strings excluded). Scripts and raw data: the session's scratchpad, lookback/. No customer identifiers appear in any file name used here.