GitHub Actions spend — Blacksmith trial and the credit runway

Where the CI bill goes, what Blacksmith measured on a real run, and three options for making the $9k GitHub credit last longer. For Fede and Gera.

2026-09-09 · Decided: Option A (Fede, 2026-09-09), PR #7437 closed; cuts pending approval · Trello qw8iq00P · PR #7437 held, not merged

Newest: Did small PRs cause the CI jam? · 30-day CI waste audit · CI fleet inspection: what is left to save · One-workflow fold: runs per PR push · CI capacity: AWS practices and how AI-scale teams cope · What others do about CI for fleets of coding agents · Job-by-job audit, September 29 · Run inspection, September 29: are we running too many checks? · Root cause, September 28

CI capacity: AWS practices and how AI-scale teams cope (2026-09-29)

On 2026-09-29 the CI fleet hit the 320 on-demand vCPU cap. The cause: the spot setting "lowest-price" found no ARM capacity, so every job fell back to on-demand. Fix #13974 merged at 19:23 MT and moved every runner back to spot within a minute. Two research passes follow: AWS and RunsOn practices, and how teams with AI-scale CI volume cope.

Night of 2026-09-29: what shipped and the proof

The fleet was capped at 320 on-demand vCPU and 512 spot vCPU. Root cause of the 3-hour stall: after a spot-quota burst error, RunsOn forces on-demand for 5 minutes, that bucket fills, and launches fail (force_on_demand_reason = spot_quota_snoozed in the worker logs). Quotas are not raised; the cure is lower peak demand and faster machines.

Before → after (RunsOn job logs, 30-min buckets, MT)

Merged live tonight

Where the machines go (2 h sample, 19:00–21:00)

About 58,000 jobs/day.

In flight (open PRs / builds)

22:30 proof (CloudWatch, one query for all windows; Denver time)

Waits met the target (slow case under 5 minutes). Cap hits did not fall because job demand nearly doubled versus the incident hour; every cut so far was absorbed by more pushes. The remaining lever is fewer jobs per push, which is what the one-workflow fold delivers.

Merged 22:00–23:00

• #14038 main superseded skip
• #14024 fold piece 2 (5 drift checks)
• #14028 Graviton4-only pools
• #14030 delta lane accepts any green ancestor
• #14041 semgrep schedule drift fix

Open, armed

• #14062 fold piece 3 (4 ElevenLabs/WS sync arms; fixed a permission line that made CI fail to load, plus a new drift test for caller-vs-callee permissions)
• #14077 approval-withdraw workflow wakes only on a failed CI run (about 4,000 of its 4,300 daily runs removed)

AWS and RunsOn practices

Items marked (inferred) or [INF] are reasoning, not stated by a source.

Top 5 actions

1. Ship PCO + wide Graviton family + t4g + cpu/ram ranges on 2cpu and 8cpu labels. 1–2 h. Repo only.
2. Upgrade RunsOn stack to ≥ v3.3.2 (fail-fast on capacity errors, OD spread across AZs, interrupted-job retry). 2–4 h. Vendor release, our CFN update.
3. Retune SpotCircuitBreaker (default 2/15/30 is too twitchy at this volume) and check AppSize. 30 min. CFN parameter.
4. Add an x64 spot twin label for arch-neutral jobs to widen spot pools. 1 day incl. migrating workflows. Repo only.
5. Quota + spot-health alarms and a one-off spot placement score check for both mixes, per AZ. 2–3 h. AWS reads/alarms only, no quota request.

How teams with AI-scale CI volume cope

No CI-cost or capacity posts were found from Cursor, Cognition/Devin, Factory, Replit, Vercel, Datadog, Canva or Airbnb. Shopify, Google, Meta, Uber and Chromium sources predate agents but are the primary literature. Items marked (inferred) are reasoning, not sourced.

Top 5 PropFlow is not doing (ranked)

1. Collapse 43 workflows into one orchestrating workflow with a path-based job graph. Effect: removes ~43 runner pickups/checkouts per PR event; plausibly 60-80% fewer workflow runs (inferred from 36k runs / 450 PRs = 80 runs/PR). Effort: 3-5 days, done in pieces. Dependency: none (update branch-protection required checks).
2. GitHub merge queue with batching (min 1 / max ~8, build concurrency capped) and full suite only on merge_group. Effect: full suite runs once per batch instead of per PR; Uber-style queues cut CI ~53%. Effort: 1 day. Dependency: GitHub setting plus merge_group trigger.
3. Per-agent CI budget: max 2 CI rounds per PR, then local-only until green; per-session concurrency cap. Effect: bounds the pushes-per-PR tail, which the median hides. Stripe model. Effort: 1-2 days (a hook in the fleet plus a counter check). Dependency: none.
4. Automatic flaky quarantine (detect by pass-on-retry, auto-skip + auto-issue). Effect: Slack took failed test jobs from 57% to <5%; for PropFlow, fewer agent fix-loops and reruns. Effort: 2-3 days in-house, or hours with Trunk. Dependency: none or vendor.
5. Remote cache for typecheck/lint/build outputs (Turborepo/Nx remote cache on S3). Effect: 85-93% hit rates published for named customers; highest value on per-push guard jobs where most inputs are unchanged. Effort: 2-4 days. Dependency: vendor or S3 self-host.

Also worth copying: treat the test selector as critical infra and load-test it at 10-20x (Anthropic).

Facts checked tonight

• RunsOn stack v3.3.1.
• SpotCircuitBreaker 2/15/30 (default); AppSize small.
• On-demand quota 320 (raise to 512 requested 2026-09-28, still open at AWS); spot quota 512.
• Fede rulings: no GitHub-hosted runners under any circumstance; no AWS quota requests.

What others do about CI for fleets of coding agents (research, September 29)

Fede asked whether we are the first to hit this. We are not, but nobody has published a before-and-after CI bill for an agent fleet. The closest is Linear, whose test suite quadrupled after agents arrived; they moved off GitHub runners, sharded more and roughly halved runner time per test. Large monorepos (Google, Uber, Stripe, Shopify) solved the same shape at bigger scale and those lessons transfer.

The local numbers that matter

Most PRs are one push. One push fans out into 40-plus workflows, and a PR fires again on the review verdict, body edits and labels: roughly 75 workflow runs per PR. Boot, checkout and install overhead on 36,000 runs a day is the structural cost.

What transfers, ranked

1. One PR workflow, not forty. One job graph that skips what the diff cannot affect. Largest lever not yet pulled; it is the redesign held back on Sept 29 and should come next.
2. Affected tests per push, everything once per merge. Stripe runs ~5% of tests per build for under a tenth of the compute (source); Shopify cut p95 from 45 to 18 min (source). Done for tripwires in #13861; the approval-time full suite is the remaining candidate.
3. Batch the merges. GitHub merge queue tests a batch of approved PRs together and bisects on red. Google reports 72% fewer machines at batch size four (source); Uber cut CI resources 53% (source). At ~400 merges a day the approval-time full suite would run per batch, not per PR. Needs required-check rewiring (branch protection: Fede).
4. No CI on non-code events. Body edits, labels and review comments start runs today; only pushes and the final verdict should.
5. The agent's machine is the first gate. Codex and Copilot coding agent run tests in their sandbox before opening a PR. Here: affected tests only, serialized through one lock, never the full suite locally (two concurrent full suites pushed the Mac mini to load 110 on Sept 29).
6. Content-hash result caching for typecheck, lint and build (Turborepo/Nx remote cache; Mercari reports ~50% CI reduction, source). Fits our build steps, not vitest itself.
7. Runner settings that cut refusals without a quota request: spot allocation by price-and-capacity instead of lowest price, a small warm pool in working hours, dependencies prebaked into the image (RunsOn, AWS). Vendor percentage claims unverified; effect unmeasured until tried.
8. Quarantine flaky tests. Google found 84% of pass-to-fail transitions are flakes (source); each flake here costs a full re-run round.

Recommendation

Today's six cuts remove about half of machine minutes. The next half is items 1, 3 and 4, in that order: the one-workflow redesign first because it is inside the repo, the merge queue second because it touches branch protection. Not proposed: raising AWS limits (Fede: never) and capping concurrent agents (an owner decision, not engineering). Status: proposed, pending Fede's go on item 1.

Research agents' full notes: merge queues and batching, selective testing, agent-fleet practices, spot runner settings (four web-research passes, Sept 29). Other sources: Linear write-up; Meta predictive test selection; Google TAP; Uber SubmitQueue; Mergify on merge queues and agents.

Job-by-job audit, September 29: what stays, what goes

Fede: "each job manually inspected: is it adding value, is it necessary; if not we're cutting to the bare bone." Measured from every job in a two-hour weekday window (Sept 28, 14:00–16:00 UTC: 3,558 runs, 9,750 jobs), scaled to a day. Session 004, 2026-09-29.

Where the minutes go (machine-minutes per day, measured)

Sources: job records for the window above (scratch file census.md, session 004); ci.yml, per-PR and scheduled workflow inspections (three reports, same session). Minutes at GitHub's ARM rate would be about $11,800 a month; the September bill is higher because GitHub rounds every job up to a whole minute and thousands of jobs run under 20 seconds. Since Sept 29 the same minutes run on our own AWS spot machines at roughly a tenth of that rate, so cutting minutes now matters most for staying under the 512-vCPU spot ceiling and for the on-demand overflow, which costs 5–10× spot.

Decisions for Fede (yes / hold, one line each)

1. Guard pins only when the push touches what it pins, plus at approval and on main. Biggest single cut, about a third of everything. Recommended: yes.
2. Cut self-review.yml. Duplicate of the pre-push hook and the Lint job. Recommended: yes.
3. Advisory harnesses run once on main after merge, not per push. Recommended: yes, with the red-main alert kept so a break pages the owner.
4. Semgrep on sensitive paths per PR, full scan weekly. Recommended: yes.
5. Require one "CI" check instead of Build, Type Check, Unit Tests, Lint. Lets the plumbing jobs fold away. Needs the branch-protection edit. Recommended: yes.
6. Nightlies weekly. Recommended: yes.
7. Redesign for AI-volume (bigger, later): one fast job per push (types, lint, affected tests, ≈8 min on one box) and the heavy set (guards, full suite) once per merge batch through a merge queue with a minimum batch of 4. Cuts the heavy tier another 60–75%. The July merge queue failed because batches of 1 and flaky tests were never quarantined; both must be fixed first. Recommended: hold until items 1–6 land and are measured for a week.

Items 1–6 are each one PR, dark for nothing (CI only, no customer effect), and reversible by revert. Together they remove roughly half of today's machine time and about 15 machine boots per push.

One-workflow fold: runs per PR push (measured 2026-09-30 02:45 MDT)

We are folding the PR-triggered workflows into a single CI run. Five of 13 pieces were merged by 02:30 (#14062, #14089, #14107, with #14111 armed and #14123 stacked; #14116 widens the runner pool). Measured, not estimated: for each PR head SHA, every workflow run with that head_sha across all events.

≈29% fewer runs per push so far; target is 60–80% once the wake list (piece 9), the docs fold (piece 7) and the regression gate (piece 8) land.

Caveats: pushes are approximated by PR commit time; machine-minutes per push are not measurable via the timing API (self-hosted runners report 0 billable ms).

Measured 2026-09-30 after pieces 7, 8, 9 and 12

A PR push now starts a median of 14 workflow runs instead of 33, measured on 20 real pushes after the fold against 25 before it.

Merged: piece 7 #14142, piece 8 #14123, piece 9 #14196, piece 12 #14127. Next: cut the guard's wake-ups (piece 13).

CI fleet inspection, September 30: what is left to save

Measured 2026-09-30. Findings only, no changes made. Each item was checked twice.

• Machine bill: $841 so far in September. Credits cover all of it. Cash spend is $0.
• Money on the table: $110 to $550 a month. Any saving makes the credits last longer.
• Time on the table: about 2 minutes off every PR push (7 min now, about 5 after the fix). That is about 100 to 1,300 waiting hours a month. It is waiting time, not paid machine time.
• Findings: 62 checked. 8 confirmed, 28 refuted, 26 dropped as unverified. The first pass claimed about $15,000 a month. Two checkers per finding cut it down. The dropped ones alone claimed $2,630 a month, more than the whole machine bill.

The 10 biggest items

Rows 3 and 4 are not counted in the totals. A higher run volume could make rows 5 and 7 several times bigger (see caveats).

The next 3 PRs

1. Rebalance the guard-test groups by measured runtime. Include a planted control: drop one file and the check must go red. First confirm the slow group is not one giant file. A fix, no switch, live everywhere.
2. Widen spot on the x86 machine type and add a weekly spot-capacity check. Never narrow the ARM choices. A third zone is a separate network design, not part of this PR.
3. Switch full-history checkouts to fetch file contents on demand. Full history stays, so finding the common base commit still works. Prove it against the planted controls.

Two structural findings

Measured today

Methods and caveats

• Every finding went to two checkers: one redid the numbers, one read the repo to see if the fix could work. Tables use the conservative figure.
• Run counts have an API ceiling. Several checks got exactly 2,500 (or 1,000) runs for 30 days. Estimates built on that may be up to 10x low. The $841 of September EC2 stays the hard ceiling on money waste.
• Samples are small: 60 PRs, 20-29 CI runs, 8 full-suite groups. Sep 29 is an outlier and was not scaled to a month.
• Hours means waiting time unless marked machine. Not measured: review model spend, why spot was disabled on 8-core, and whether the slow group is one heavy file.
• The source report says 60 findings and Sep 29 at about $236; this page uses 62 findings and $243 as given for publishing.

30-day CI waste audit: where time and money go

Audit — 2026-09-30, partial coverage. Read-only, no changes made.

Coverage

This is a sample, not the full month. We read 210,500 runs out of an estimated 700,000 to 750,000, and 2,909 merged PRs out of an estimated 4,000 to 4,500. We have no per-machine minutes, because the GitHub API budget ran out. Dollars come from the AWS bill.

Where a PR's time goes

Based on 80 random merged PRs. For the median PR, getting live takes longer than getting merged.

Outcomes of runs: 38% were skipped with no work done, 6% were cancelled, and cancelled runs used 23% of CI time.

The 10 biggest waste items

Code review waste

• 27% of review runs give no verdict (20% fail, 7.5% are cancelled).
• 71% of suggestion-only verdicts were followed by a push.
• Reviewer turn budgets are 3 to 5 times what is used, and no review hit its cap.

Corrections

The approval-time rerun already skips when the PR's unit tests concluded (G5 in scripts/ci/decide-test-lane.sh), so the 45% figure is a run count. The machine-time share is being measured separately.

Did small PRs cause the CI jam? — 2026-09-30

Brief — 2026-09-30. Read-only, no changes made. Figures marked estimate are estimates.

Verdict

Partly, and only a little. The jam comes from how many PRs our parallel sessions open at the same moment, times a big fixed cost every push pays no matter its size. The size rule adds maybe a few percent on top (estimate).

• In the 6 busiest hours of Sep 28-30, CI saw 46 to 81 different PRs per hour, with only 1.2 to 1.6 pushes each. The pile-up is many PRs at once, not the same PR pushed over and over.
• Each push costs about 55 machine-minutes (estimate, 15 pushes measured job by job), and size barely moves it. The side checks cost about 12 minutes whatever the size. Small PRs were not cheaper than big ones: median about 60 minutes for PRs up to 300 lines, about 32 for PRs over 600.
• Big PRs are where the defects are. PRs over 1,000 lines averaged 9 pushes and 8.6 review rounds, and 7 of 13 got a blocking finding. PRs up to 300 lines averaged about 2 pushes and 1.3 rounds, and 0 of 29 got one.

Options

Recommendation: A. Keep the size rule. It is the cheapest defect filter we have. Put the effort into making each push cheaper and into pacing bursts. A and C can stack later.

Per PR by size

Based on an 80-PR review sample plus a 1,303-PR join to the sampled run list. The run list keeps only the newest 1,000 runs per 3 hours, so push and run counts are undercounts; the ratios between sizes hold.

Measured: runs per push, before → after

Machine-minutes are not comparable yet: the after figure (about 62 per push) was measured job by job, but the before figure was not, because the API budget ran out.

• Still firing on nearly every push (next fold targets, about 10 per push): Auto-merge on green, Branch Preview Domain, CI Dispatch Guard, Nested Package CI, Claude Code Review, Claude Code, Review execution re-pin, Claude Review Check Override, Claude Review Recheck, and the no-op review debounce.

Side findings

• 381 of 1,403 sampled jobs (284 machine-min) ran on GitHub's own ubuntu-24.04-arm machines, mostly CI jobs. That conflicts with the rule against GitHub-hosted runners. Worth a check.
• The Blacksmith bot (blacksmith-sh[bot]) triggered 83 runs in the burst hours. Unexplained.
• The same head commit got two pull_request CI runs 18 seconds apart (#14282); one cancelled the other. A duplicate-run-per-push defect to fix after the freeze.

Run inspection, September 29: are we running too many checks?

Documented, no changes made to the checks yet. Decisions pending Fede. Session 004, 2026-09-29.

Six changes merged today, every run followed

A one-push change used about 25 runs and 40 machines. A change with 3-4 pushes used about 100 runs and up to 165 machines.

Decisions needed from Fede

1. Approve the order: narrow triggers first (A), combine tiny gates second (B), fewer pushes per change third (C). Recommended: yes.
2. Retire checks that have not failed in the sample (D)? Recommended: not yet, one check at a time.
3. Act now, or wait for the first full day on our own AWS machines (2026-09-30)? Recommended: start A now, since it does not depend on the measurement.

Next three actions

1. Gera's swarm (owner of the check settings) narrows what wakes the helper jobs (A).
2. Session 004 combines the tiny per-change gates into one job (B).
3. Read the first full AWS cost day on 2026-09-30 and report before and after.

Why this came up

At the 2026-09-08 standup Fede said GitHub Actions is costing thousands a month. Aldo (ex-colleague, at Zillow) said he went from about $300 to $80 a month on Blacksmith and that it is faster. Fede ran Blacksmith's migration wizard on 2026-09-09; it opened PR #7437 swapping the runner label in all 66 workflow files. The goal Fede stated in session: about $9,000 of GitHub credit remains, it expires in twelve months, the run rate is over $3,000 a month, and the aim is to make the credit last longer, not to hit twelve months. Cash spend on a vendor is only worth it if speed matters somewhere.

Prior work on the same bill: a self-hosted runner on the Mac mini was tried and reverted on 2026-07-31 (Fede: no self-hosted runners). The August 9 to 11 sessions targeted a 50% cut and shipped push debouncing, incremental re-review and guard consolidation; the target was not reached. On 2026-08-18 the full test suite moved to GitHub's 8-core ARM pool (PR #5856), measured then at 7 min to 2.3 min wall and $0.35 to $0.22 per run. Blacksmith itself had never been evaluated before today.

The bill, measured

Source. Every job GitHub ran between 2026-09-08 23:28 UTC and 2026-09-09 17:04 UTC, pulled from the Actions jobs API: 2,982 workflow runs, 10,752 jobs, of which 4,395 actually executed (the rest were skipped by an if: and are not billed). Priced at GitHub's published private-repo list rates per runner label, each job rounded up to a whole minute as GitHub bills, then scaled to 30 days. These are modeled figures, not the invoice; the org billing endpoint needs an admin scope this session's login lacks.

What Blacksmith measured on PR #7437

Source. The PR's own checks ran on Blacksmith machines because a workflow edit takes effect inside its pull request. First run at 2026-09-09 17:44 UTC, a rerun of five workflows about an hour later after the first run had warmed what it could. GitHub-side numbers are the median of the same job over the sample above, and step timings come from one representative GitHub PR run (34379781500) against the Blacksmith rerun (34384721571).

Every check that matters passed on Blacksmith: build, lint, type check, all unit test shards, the leasing regression gate, the escape gate, the preview deploys, the harnesses. The one red check, the renewal conversation scoping eval, fails identically on main today (dispatched on main at 17:56 UTC to confirm) and is a prompt problem unrelated to runners. It is not a required check.

Pricing, side by side

Source. blacksmith.sh/pricing and docs.blacksmith.sh runner overview, fetched 2026-09-09; GitHub rates from the 2026-08-18 billing pull recorded in the CI workflow file and the August sessions.

Aldo's $300 to $80 result is consistent with a small team whose jobs are compute-bound and whose GitHub bill was mostly larger-runner minutes. Our bill is thousands of one-minute jobs, where the rate is identical.

Options

Cuts that extend the credit (Option A)

Ranked by modeled monthly saving from the same sample. Arithmetic: jobs in the 17.6 h sample x 1.364 for a day x 30 x billed minutes x rate. All are estimates until shipped and re-measured.

The mechanical cuts alone are about $370 a month, 13% of the bill, and take the credit from roughly three months of runway toward three and a half. Getting to a real stretch needs the two design decisions above (review parking, full suite on verdicts), which together are another $500 or so and would push runway toward five months.

Decided 2026-09-09 (Fede)

Shipped 2026-09-09

Measured after the cuts (2026-09-10)

On that last row — the red-main count needs unpacking, not just quoting. Grouping the 58 red pushes into distinct incidents (not counting the same broken thing twice while it's being fixed forward) gives 14 separate episodes, not 58. Together they left main red for about 9.2 hours total across the 15.3-hour window (the longest single stretch was 1.9 hours, right after the cut shipped) — that's the time production promotes were held.