CI/CD architecture: from opening a PR to proven in prod

Shipped today

In flight

All ideas, by status

Today's numbers: queue peaked at 414 runs waiting (1:20 PM), 158–232 through the afternoon; 41 PRs merged 2:30–5:00 PM; main green since 5:04 PM; at 5:25 PM the merge queue held 20 PRs (18 Gera's) with the first 10 running as a batch; fleet 170 machines. 5:30–6:10 PM: 97 PRs merged after the queue came off (vs 4 merges 4–5 PM). 6:35 PM: Fede's 37 open PRs: 10 armed green/awaiting review, 26 red or conflicting, builders assigned; 9 of his merged in the following hour. Production web still at the 5:01 PM build at 7 PM because every main run since failed the one tour-wording test; deploys fire only on a green main. 7:50 PM: Fede's open PRs 24 (from 38 at 6 PM); 41 PRs merged in the previous hour; production on the 7:00 PM build; 108 spot machines running, 142 jobs queued; 90 spot no-capacity errors in the last hour. 8:35 PM: main red 7:00–8:22 PM on guard tests, fixed forward with three PRs; production still on the 7:00 PM build; 22 PRs merged in the previous hour; Fede's open PRs 26; 133 machines running, 160 queued; ~17 orphan machines at any time (job cancelled before pickup, killed at 10 min, not tunable).

Every gate from laptop to prod, colored by what we should do with it. Numbers: 25 pushes, 551 runs, fetched today.

Seven checks are required to merge (Build, Type Check, Unit Tests, Lint, Leasing regression, review, Nested Gate). Everything else is extra. Already removed from the push path today: PR body gate, file collisions, size tier, docs fence, Semgrep, Quality Gate POC, Dependabot gate, lambda bundle check.

Review loops, not CI speed, eat the hours. Top bar: 600-PR sample. Bottom: median PR timeline.

34 failed main CI runs, logs read; one run can have several causes.

Every test file classified (heuristic script plus spot reads); 899 files to merge or delete.

The 546 drift tests: most guard a regression that really happened, so they stay.

The rule: a test earns its place if it checks behavior with real inputs, has caught a real defect, or guards a stated rule. Delete tests of run-once scripts, duplicates, and ones that only echo mock calls. The check: each new test carries a mutation control (plant the bug, see it go red), and a nightly report lists tests with zero catches in 90 days, flaky ones, and test-to-code line growth.

Winning design: reliability-first (2 of 2 judges), plus grafts from the other two proposals.

Kept: the dangerous-diff hold, the required check names (published on both PR and queue events), the Willows-only probe. Not adopted: a cheaper review model on quality paths, or moving drift tests to nightly.

2026-10-01 20:27 UTC: merge queue turned OFF on Fede's emergency go (depth 71, "blocking all of us"). Before the flip, kick-red-out (#14934) had 27 of 32 batches passing and 0 failing; the 20:03 UTC ruleset drop that landed it had flushed and disarmed every queued PR. PRs now merge one by one with plain auto-merge; main-red collisions are the risk, the deploy hold is the backstop. Turning it back on is Fede's call, after #15019/#15043/#15052 land and the pre-queue check is required.

1. in flight: #14289 armed
2. retry live (watcher + rerun workflow); one runner spec not started (163 copies)
3. done: #14323 merged 2026-09-30
4. not started
5. yes from Fede, after catch-up; not started
6. in flight: pieces 14 (#14267), 16 (#14354) open; 17 built
7. not started
8. not started

Green row: reliability first, each step reversible. Blue row: flow changes, each deletes gates.

before-after-runs-per-push-2026-09-30.md · ci-gates-inventory.md · ci-waste-audit-30d.md · deploy-post-merge.md · judge-0.md · judge-1.md · pr-lifecycle.md · proposal-0.md · proposal-1.md · proposal-2.md · research-best-practices.md · review-process.md

Disputed numbers: the $6.6k/month runner figure is one expensive day (Sep 29) times 30; capacity and red-main targets are queue math, not measurements. Not measured yet: change failure rate, recovery time, review token spend.

Fede asked, 2026-09-30: Gera posted a Buildkite link in the meeting chat ("our high volume has broken the build machines" / "and our wallet" / buildkite.com). Would a vendor be cheaper than maintaining our own fleet? Base: ~1.7M machine-minutes in September (spend page).

Recommendation: keep the fleet, finish the CI cleanup (the savings are in minutes, not the vendor), revisit only if stalls keep costing hours. Every vendor cell is list price × our minutes (assumes 1 machine-minute = 1 billed 2-vCPU minute); "after cleanup" assumes the team's own 50% target, not a measurement.

Sources: buildkite.com/pricing, buildkite.com/docs/agent/buildkite-hosted/linux, buildkite.com/docs/pipelines/migration/run-github-actions-workflows, blacksmith.sh/pricing, depot.dev/pricing, runs-on.com/pricing, docs.github.com/en/billing/reference/actions-runner-pricing, github.blog changelog 2025-12-16 (simpler Actions pricing). Transcript: #transcripts 2026-09-30 chat log and call.

One machine per PR push (Proposed — pending review, Oct 1, 9:40 AM MT)

Answer. Run everything a PR push needs on one 8-core arm spot machine as steps of one job, and have that job post the required check names itself. Today a push starts about 25 short glue jobs on the always-on pool plus 3–4 heavy machines; the glue jobs and the machine boots cost more than the work. Cost model (Opus, Oct 1): CI runs about $335/day against Fede's target of under $65; this piece is about −$68/day and takes push-to-green from 9.8 to about 8 minutes.

What breaks, and the answer.

1. Required check names. Branch protection accepts a check-run of that name from the Actions app; the job needs checks: write. The rule "execution jobs post nothing" (one chokepoint) is kept: a single final step posts, and the existing chokepoint mutation controls stay green.
2. Re-running one check. One job means one rerun (~8 min) instead of one shard. Accepted: reruns are for machine kills, and the spot-kill rerun already restarts the whole run.
3. Follow-ons keyed on job names. The review watchdog and the merge guard read the job "Unit Tests" and its "CI verdict:" step; the guard is deleted after #14830, the watchdog moves to reading the check-run. Phase 2, before any required name moves.
4. 145 tests reference ci.yml; 15 pin names and the job graph. The shadow adds a new job under new names and touches none of them; the cutover PR updates them with the controls.

Plan. (1) Shadow, dark: job "One machine (shadow)" runs on 1 in 4 pushes (head sha starts with 0–3), posts its results as shadow: check-runs nobody requires, records step timings; a dated removal PR 48 h after merge. (2) Compare on the sampled pushes: shadow wall time vs the real lane's push-to-green, red/green agreement per name. (3) Cutover: the shadow posts the required names, the old jobs go, tests and controls updated, follow-ons re-keyed. (4) Glue that stays outside the push (reviews, merge-queue verdicts) moves to two always-on 2-core arm machines (~−$18/day, removes the warm pool).

Cost arithmetic (Opus, model3.py → one-machine.md, measured 12:00–14:45Z Oct 1 after the queue went on, scaled to a day, ±25%).

Unmeasured, and what the shadow measures: the 70% CPU efficiency and the 90 s setup. Open: GitHub expressions have no arithmetic, so sampling uses the head sha's first character. Status rows below as PRs land.

Where the minutes go (measured 2026-10-01)

Gera's message on 10-01: CI still feels slow. We measured where a PR's minutes actually go, on the PR lane and in the merge queue, before choosing the next lever.

Measured 2026-10-01 10:20–11:40 MT by the CI/CD Fable session; samples: 40 PR runs, 286 batch runs, 12 failure logs, 8 batch critical paths, 39 PRs.

PR lane (last 3 h, 40 runs, 14:20–17:20Z)

Merge queue (since it turned on 10:49Z; 286 batch runs, 39 merged PRs sampled)

What this says.

• The fleet is not the bottleneck: machine pickup is seconds, and the review bot finishes first.
• The felt slowness is the merge queue: a PR waits about half an hour, half of it just waiting for a batch slot.
• Three levers, by owner: queue settings (more batches built in parallel) — CI/CD Opus; CI Checks taking 9 min inside a batch when it takes 3.4 on a PR — CI/CD Fable, cause being traced now; drift pins that only run in the queue — a re-check against fresh main right before enqueue, so a stale-green PR never enters the queue (proposed to Opus).
• Not in scope here: the slowest-tests audit, which Gera is doing himself.

One run per PR push: baseline and plan (Oct 1, 5:30 AM MT)

Measured on real PRs, last 3 hours (98 pushes). Each PR push fires 3 workflows directly and about 5 more when CI finishes, roughly 9 runs per push.

Target: 1 direct run per push and 1 follow-up run per CI completion (from ~9 to 2), measured the same way after each step. Machine-minutes change little in parts 1 and 2 (same jobs, fewer runs); the review-as-job step is where the pickup waits disappear.

Before → after, measured the same way (2026-10-01)

Method: every workflow run in the window (events pull_request, workflow_run, repository_dispatch, pull_request_review, pull_request_target) divided by distinct PR pushes. Before = 02:00–05:00Z (merge queue OFF). After = 12:50–14:50Z (merge queue ON).

Not a win yet, and why. The five CI-completion helpers are gone (3.7 runs/push → 0) and replaced by one file (4.8/push). The direct runs fell 3.6 → 2.8. But two things grew: the auto-merge guard now fires on every CI completion (5.9/push, was 4.2) and the follow-ups file fires on every CI completion too. The queue was off in the before window; with it on, every merge-queue batch's CI completion fires both workflows with every job skipped, and the after window had fewer pushes per queue batch, so the ratio rose. The leg ci.yml runs on every review (approval-time full suite, 1.5/push, 46 runs) and the merge-hold review listener (1.5/push, job skipped on bot reviews) are new in the count because the before tally missed them.

Re-measured 10:00 AM MT (45 pushes since the queue-branch filter merged at 9:10): 16 runs per push, down from 18.3. Where they come from now, per push: follow-ups file 4.3 (it wakes on every completion of CI, the review, harnesses and advisory tests, almost always with every job skipped); the auto-merge guard 3.2 on completions plus 1.4 on reviews; merge-hold listener 1.4; ci.yml 1.3 on reviews (the full suite at approval) and 1.3 on pushes; the review itself 1.0; dispatch guard 1.1; deploys after merges 0.3. The queue-branch filter worked: zero follow-ups now fire on merge-queue batches.

What this says. Every listener that wakes on a completion is one run per completion, no matter how little it does; filters only trim which completions. Deleting the guard (after the fence PR) takes ~4.6 off and the review-event suite cut takes ~1.3, landing near 10. Getting to 2 needs the producer to do its own follow-up work: the review as a job inside the push's run and the follow-ups as its final steps, which is what the one-machine-per-push design above delivers. Target stays 2; re-measured the same way after each step.