Which tests fail most, and which take longest

Fede's page lists a flaky-test ranking as "done, list in session". That list was never published, so this page rebuilds it from the logs. It is consistent with his 34-run sample: drift pins, led by the route inventory, are the repeat offenders here too, and maintenance-comms-activities from his list shows up here as well.

How to read the count. Main runs CI on every merge, about 180 times a day. When a test breaks on main it fails every run until a fix lands. So "failed 70 times" usually means "stayed broken for 4 hours", not "fails at random". The last column shows which kind it is. A one break needs a faster fix-forward. A repeat is a quarantine candidate.

Quarantine candidates: files that broke main more than once

These are the ones a flaky-test quarantine would actually catch. A break is counted again when the test fails after more than 3 hours of passing.

Four of these call live ElevenLabs and compare it to the repo (specialist-prompt-drift, live-agent-roster, specialist-transfers-drift, live-equals-main). They go red whenever someone edits an agent in ElevenLabs, whatever the code says. That makes them better suited to a nightly alert than a merge gate. user-route-inventory is the single worst offender: 6 breaks in 7 days, because every new page route has to be added to its pin.

Average is the mean across 25 complete runs on Oct 1. Per test is the file's average divided by its number of tests: a high value means a few heavy tests, not a big file.

The single slowest tests

The slowest individual tests are mostly waiting, not working. The top one is the whole of appfolio-client.test.ts's 68 seconds: it waits through real retry delays instead of a fake clock.

vitest only prints a single test's time when it is slow, so this table is the slow tail, not every test.

Every file in the 52 reviewed (the 40 slowest plus every repeat failer) had its code read and was given one suggestion. Files further down the list say not reviewed. Seconds are each label's share of the average running time.

Baseline: the numbers to beat (Oct 1, 2026)

Re-run the same measurement after the fixes land and compare against these.

Re-measure after rows 82 and 40 (Oct 3, 15:47 UTC, main 60791eee54e)

Each baseline figure is the median of the Oct 1 runs on this machine type (4 runs for half 1, 5 for half 2). Across their whole range the change is −34% to −37% (half 1 job), −37% to −41% (half 2 job), −47% to −54% (loading) and −35% to −42% (running).

What row 82 actually saved (vitest's own per-module import self time, both halves, the throwaway reporter: Oct 2 run 37027978394 on m7g.2xlarge, Oct 3 run 37133264469 on m8g.2xlarge). The two machines differ, so Oct 3 is also shown at m7g speed. Across 14 heavily loaded modules nobody changed in between, m8g took 0.73 times as long per load (middle half 0.68 to 0.77).

All first-party module self time: 4,159 s on Oct 2 (m7g), 2,688 s on Oct 3 (m8g), which is about 3,670 s at m7g speed (−12%).

What is left, biggest first. src/lib/platform/auth/server.ts 519 s over 1,208 test-file loads on m8g (about 710 s at m7g speed; it was 660 s over 1,011 on Oct 2, because more files reach it every day). Trimming it is an auth change, so it needs its owner. src/lib/domain/leasing/lease-terms/dates.ts 112 s, now the first loader of date-fns/format; the same lightFormat change would likely remove most of it, unless the cost moves again. src/lib/domain/vendors/vendor-calendar.ts 33 s, the first loader of date-fns/parse.

Basis and limits. Timing: run 37133262569 (ci.yml dispatched on main at 60791eee54e), both halves on m8g.2xlarge, 3,994 + 3,994 test files, all passing (3 + 6 skipped). Baseline: this page's Oct 1 runs on m8g.2xlarge (half 1: 36829783241, 36882040216, 36875447902, 36870896634; half 2: the same four plus 36872698236). The Oct 1 file count is that day's 3,769 per half. One timing run, so the yardstick for noise is the baseline's own spread (886 to 926 s for half 1's job). Breakdown: run 37133264469, the same tree plus the throwaway reporter on a branch deleted afterwards. One test, self-review-unresolvable-range, failed only in that run; the plain run had none. Row 40 (the watchdog test leaving the per-PR lane, about 43 s of one worker) is too small to see in a half's wall clock, and rows 83 and 84 only add mocks.

Final re-measure (Oct 2, 16:01 UTC, main 16206803b2a)

Against the Oct 1 medians above (mixed machines, mostly the faster Graviton4), today's runs on the slower Graviton3 still come in under every one: one half's whole job 16.7 → 12.5–13.3 min, vitest wall clock 14.6 → 11.8–12.6 min, loading 62.3 → 44–47 worker-minutes, running 24.0 → 18–21 worker-minutes. Half 2 today: 12.5–12.7 min job, 709–719 s vitest; its Oct 1 runs were all on Graviton4 (842–933 s), so there is no same-type pair for it.

Where the loading went (vitest's own per-module import self time, both halves, two runs with the throwaway reporter, both on m7g.2xlarge: Oct 2 01:40 UTC run 36949936682 against the final tree, run 37027978394):

What is left, biggest first. src/lib/platform/auth/server.ts 660 s over 1,011 test-file loads (it was 305 s over 475: new auth code today doubled the files that reach it); trimming it is an auth change, so it needs its owner. src/lib/utils/explicit-year-day.ts 646 s and src/lib/domain/leasing/renewal-rent.ts 320 s: the date-fns barrel is gone, but date-fns/parse (about 190 ms cold) and date-fns/format (about 113 ms, it brings the locale) still reach about 3,000 test files through them (row 82).

Basis and limits. Final: run 37027723475 (ci.yml dispatched on main at 16206803b2a) and run 37028316163 (an empty commit on the same tree, row 79’s change run), every half on m7g.2xlarge, 3,885 + 3,882 test files all passing in both. A third run on the same tree with the throwaway reporter (37027978394) fell inside that range: 725 / 720 s. Baseline: run 36872698236 half 1, the only Oct 1 baseline half on m7g. The gain is the whole effort since the baseline (every merged row on the tracker), not only the loading rows. One Oct 1 half on this machine type is a small sample, and the same tree varied by up to 5% from run to run today, which is why the table gives ranges.

Where the loading time goes (Oct 2)

Interim re-measure (Oct 1, 23:33 UTC, main 4d1921089)

Basis and limits. Interim: run 36941383284 (dispatched on main; 6 + 3 test files failed from main's own reds, which does not change the timing). Each baseline row is the one baseline run on that machine type (36872698236 half 1 on m7g, 36865074180 half 2 on r8gn), so loading and wall clock within about ±8% are noise. The running-time drop is the same in both halves on two different machine types, and it is measured on the files present in both runs (the suite gained 121 test files and lost 65 since the baseline). Rows still open when this ran (20, 21, 24, 28–41, 44, 49, 75 and others) are not in it. The final re-measure comes after they land.

• Failures: every completed ci.yml run with conclusion failure on push-to-main, merge_group and schedule, created Sep 24 UTC onward. For each run, the log of every failed Unit Tests (affected) / Unit Tests (full N/2) job was read for vitest's FAIL lines. Blind spot: tests that run in the Guard pins jobs, and the database-safety check in CI Checks, are not read; see the correction in section 1. A test counts once per run. Pull-request runs are left out on purpose: a red PR is usually the PR's own bug.
• Durations: the 25 newest runs where the whole suite ran (both Unit Tests (full 1/2) and (full 2/2); 24 on main plus 1 nightly). Per-file times are vitest's own (one line per file in the log). The two halves split the suite with no overlap, and the per-file times add up exactly to each log's own total.
• Limits: all duration samples come from one day, on the fleet's mixed spot machines, so the absolute seconds will vary. The order is stable: medians track the averages. Job logs are kept about 90 days, so the failure window could be widened.