GitHub Actions spend — Blacksmith trial and the credit runway

Where the CI bill goes, what Blacksmith measured on a real run, and three options for making the $9k GitHub credit last longer. For Fede and Gera.

2026-09-09 · Decided: Option A (Fede, 2026-09-09), PR #7437 closed; cuts pending approval · Trello qw8iq00P · PR #7437 held, not merged

Blacksmith does not lower the bill, it moves it from credit to cash. Its 4-vCPU machines cost the same per minute as GitHub's standard runners, both round every job up to a whole minute, and about 4,400 billed jobs a day are the problem, not the hardware. Recommendation: stay on the credit, close the PR, and cut job count.

Why this came up

At the 2026-09-08 standup Fede said GitHub Actions is costing thousands a month. Aldo (ex-colleague, at Zillow) said he went from about $300 to $80 a month on Blacksmith and that it is faster. Fede ran Blacksmith's migration wizard on 2026-09-09; it opened PR #7437 swapping the runner label in all 66 workflow files. The goal Fede stated in session: about $9,000 of GitHub credit remains, it expires in twelve months, the run rate is over $3,000 a month, and the aim is to make the credit last longer, not to hit twelve months. Cash spend on a vendor is only worth it if speed matters somewhere.

Prior work on the same bill: a self-hosted runner on the Mac mini was tried and reverted on 2026-07-31 (Fede: no self-hosted runners). The August 9 to 11 sessions targeted a 50% cut and shipped push debouncing, incremental re-review and guard consolidation; the target was not reached. On 2026-08-18 the full test suite moved to GitHub's 8-core ARM pool (PR #5856), measured then at 7 min to 2.3 min wall and $0.35 to $0.22 per run. Blacksmith itself had never been evaluated before today.

The bill, measured

Source. Every job GitHub ran between 2026-09-08 23:28 UTC and 2026-09-09 17:04 UTC, pulled from the Actions jobs API: 2,982 workflow runs, 10,752 jobs, of which 4,395 actually executed (the rest were skipped by an if: and are not billed). Priced at GitHub's published private-repo list rates per runner label, each job rounded up to a whole minute as GitHub bills, then scaled to 30 days. These are modeled figures, not the invoice; the org billing endpoint needs an admin scope this session's login lacks.

Item	Per month (modeled)	Share	What it is
Full unit-test suite, 12 shards on the 8-core ARM pool	$647	23%	About 57 full-suite runs a day. 54% of the shards are triggered by review verdicts on hub-module PRs, 27% by pushes to main, 17% by PR pushes that escalated to full mode.
Type check, build, lint on every PR push	$430	15%	Three jobs per push, roughly 120 to 180 of each a day.
Affected unit tests on PR pushes	$257	9%	One shard per push, median 56 s.
AI code review runner, parked while the model reviews	$167	6%	Median 5.1 min, p90 8.6 min per review, on a billed machine that does nothing but wait.
Guard pins	$113	4%	Post-merge only since 2026-08-20; still 229 jobs a day.
Everything else, about 60 small jobs	$1,225	43%	Gates, drift checks, preview mint and remove, harnesses, evals, deploys. Most finish in under 30 s and bill one minute each.
Total	$2,839	100%	Matches Fede's "over $3k" once storage and the review bot's own spend are added.

Round-up is about $800 a month of the total. Raw seconds price out at about $1,990 a month; whole-minute billing makes it about $2,840. 2,394 executed jobs in the sample finished in 30 s or less and each billed a full minute.
Volume, not machine size. About 4,000 workflow runs a day. The 8-core pool is already the cheapest way to run the suite per run; the cost is how many times it runs.
Correction to a first read of the data: the Dependabot review gate and the CI dispatch guard looked expensive in a per-workflow rollup but are not. Both skip in seconds on 96% of runs and cost a few dollars a month.

What Blacksmith measured on PR #7437

Source. The PR's own checks ran on Blacksmith machines because a workflow edit takes effect inside its pull request. First run at 2026-09-09 17:44 UTC, a rerun of five workflows about an hour later after the first run had warmed what it could. GitHub-side numbers are the median of the same job over the sample above, and step timings come from one representative GitHub PR run (34379781500) against the Blacksmith rerun (34384721571).

Step or job	GitHub	Blacksmith	Read
Type check, the CPU-bound step alone	79 s	35 s	2.3x faster Twice the cores show up where the work is pure compute.
Lint, the CPU-bound step alone	5 s	7 s	Same.
Hermetic harness jobs (AppFolio loop, turnover intake, app intake, portfolio), whole job	116 to 174 s	68 to 79 s	40 to 58% faster One run each on Blacksmith; consistent direction.
Semgrep scan, whole job	77 s	47 s	39% faster
Installing dependencies	29 s cache restore	90 s full npm install	3x slower, every job Main seeds the node_modules cache on GitHub's cache backend. Blacksmith redirects the cache action to its own backend, so nothing seeded on GitHub is visible there. This persists until the seed job itself runs on Blacksmith, which means a full move, not a partial one.
Git checkout	5 to 25 s	13 to 288 s	Unstable One checkout took nearly five minutes; most took 13 to 26 s.
Lint, whole job, first run	49 s	629 s	Cold lint cache plus cold dependencies. Warm rerun: 141 s, still 3x slower because of the install.
Affected unit tests, whole job	109 s	74 s, then 533 s on rerun	Inconclusive The rerun selected a different set of tests and hit the slow checkout.
Small gate jobs (nested gate, quality gate, preview mint)	14 to 17 s	55 to 105 s	Slower These jobs are almost all install time. On GitHub the install is a 29 s restore; on Blacksmith it is a full install.

Every check that matters passed on Blacksmith: build, lint, type check, all unit test shards, the leasing regression gate, the escape gate, the preview deploys, the harnesses. The one red check, the renewal conversation scoping eval, fails identically on main today (dispatched on main at 17:56 UTC to confirm) and is a prompt problem unrelated to runners. It is not a required check.

Pricing, side by side

Source. blacksmith.sh/pricing and docs.blacksmith.sh runner overview, fetched 2026-09-09; GitHub rates from the 2026-08-18 billing pull recorded in the CI workflow file and the August sessions.

Machine	GitHub	Blacksmith	Note
Standard x64 (GitHub 2-core vs Blacksmith 4-vCPU, the wizard's mapping)	$0.008/min	$0.008/min	Same price, twice the cores. Blacksmith bills per vCPU: $0.004 per 2-vCPU minute.
Standard ARM (GitHub 4-core ARM vs Blacksmith 4-vCPU ARM)	$0.005/min	$0.005/min	Same. ARM is 0.625x the x64 rate on both sides.
8-core ARM (the test-suite pool)	$0.014/min	$0.010/min	29% cheaper per minute. This is the only line where Blacksmith's rate is lower.
Free tier	Team plan minutes, not applicable to larger runners	3,000 2-vCPU x64 minutes a month	About $12 of value at our volume.
Rounding	Up to the minute	Up to the minute	The $800 a month of round-up moves with the jobs.

Aldo's $300 to $80 result is consistent with a small team whose jobs are compute-bound and whose GitHub bill was mostly larger-runner minutes. Our bill is thousands of one-minute jobs, where the rate is identical.

Options

Option	GitHub credit burn	Cash	Speed	Verdict
A. Stay on GitHub, close the PR, cut job count	$2,840 today, falling with each cut below	$0	Unchanged	Recommended Every dollar stays on the credit; the cuts are the only thing that actually extends runway.
B. Full move to Blacksmith (the PR as written)	$0	about $2,600 modeled at the same job mix and durations	Compute steps about 2x faster; installs and checkouts slower or unstable; net wall-clock unproven	No Trades credit for cash with no bill reduction. Also moves the cache backend, so every job re-seeds until main has run there.
C. Partial: only the 8-core test shards (narrow the PR to that one label)	minus $647	about $460 modeled at today's shard durations	Shards likely faster once the cache is seeded on Blacksmith; unmeasured, since full mode did not run on the PR	Only if speed matters Cash for credit again, and the node_modules seed would have to run on Blacksmith too or every shard pays the 90 s install.

Cuts that extend the credit (Option A)

Ranked by modeled monthly saving from the same sample. Arithmetic: jobs in the 17.6 h sample x 1.364 for a day x 30 x billed minutes x rate. All are estimates until shipped and re-measured.

Cut	Per month	Shape
Fold the auto-merge arm-or-disarm job into a step of an already-running job	$99	412 one-minute jobs a day that do 8 s of work. Mechanical.
Fold the two unit-test reporter jobs into the last shard's step	$112	465 one-minute jobs a day, 3 to 9 s of work each. Mechanical.
Gate or retire the Quality Gate POC's three jobs	$97	402 one-minute jobs a day. Needs an owner's call on whether the POC is still an experiment.
Fold preview-domain mint and remove into steps	$62	259 one-minute jobs a day.
Stop parking a runner for the AI review	up to $167	Re-architecture: the review runs inline on a billed machine for 5 to 9 min. The August 9 session already proposed an async executor on Lambda as "Decision 3", not committed.
Full suite on review verdicts	up to $350	54% of full-suite shards come from review verdicts on hub PRs, by design since PR #5518 so hub changes get full coverage before merge. Cutting this is a coverage decision, not a cleanup.
Remaining sub-30-second jobs	$125 to $165 realistic	About 1,270 more tiny jobs a day across 40 workflows; maybe a third can fold or path-filter. Low confidence.

The mechanical cuts alone are about $370 a month, 13% of the bill, and take the credit from roughly three months of runway toward three and a half. Getting to a real stretch needs the two design decisions above (review parking, full suite on verdicts), which together are another $500 or so and would push runway toward five months.

Decided 2026-09-09 (Fede)

Option A. Stay on the GitHub credit; PR #7437 closed. Approved for one PR, shipped dark through CI and re-measured the next day: move the 112 x64 jobs to GitHub ARM runners (est. $340/mo), merge type check, lint, build and the escape gate into one job per PR push (est. $230/mo), and fold the four one-minute helper jobs into steps (est. $370/mo). Deferred: full suite only on pushes to main (est. $460/mo) because it trades money for red-main incidents and would leave hub-sized PRs with no unit tests before merge; revisit after the safe moves are measured.

Shipped 2026-09-09

PR #7451 merged 20:04 UTC: 110 jobs moved from x64 to GitHub ARM runners. Three jobs stay on x64 with a dated reason each: the Playwright browser job, the Semgrep scan (its Alpine container cannot host GitHub's JavaScript actions on arm64), and the review bot's own job (arm64 boot unproven; deferred to its own PR). The review bot caught one real defect before merge: the Lambda deploy workflow's 17 dependency-cache keys lacked the architecture, which would have restored x64 build tools onto ARM and broken every production Lambda deploy; fixed in the same PR by adding runner.arch to the keys. Two free skips added for Agent Smith's nightly branches in the preview-domain workflow.
Job merge (one job per PR push): PR #7454 merged 21:17 UTC. Type check, lint, the escape gate and build now run as one job with one checkout and one dependency restore; each section keeps its own verdict so authors still see every failure, and a setup failure fails all three required names closed. Five review rounds: the bot caught that the first version would have stopped production web deploys (a skipped deferred build read as failure), posted required statuses on the wrong commit, and reported passing on setup failures; then that Dependabot and fork PRs lost their producer for the three required names (fixed with a companion workflow that runs only for bot and fork PRs). Est. about $230/mo by the model above.
Folds not done: the preview mint/remove jobs already skip each other unbilled; the arm-or-disarm no-op needs an API call a job-level if: cannot make; the two unit-test reporter jobs are kept separate by an incident rule from 2026-08-02. The $370 estimate for this line is withdrawn.
Red main, checked: 36 failed main runs 09-07 to 09-09; 32 are post-merge guard-pin checks (post-merge only since 2026-08-20), 3 Vercel-side deploy errors, 1 runner shutdown. None were full-suite unit test failures, so the approval-time full run was not what kept main green; the guard-pins cost move traded minutes for post-merge red.
Full suite only on main: PR #7461 merged 22:27 UTC. The 12-shard suite runs on pushes to main, the nightly, and manual runs only; never on a PR or a review verdict. Affected tests still run on every PR push; a PR whose selection exceeds the cap reports "N affected files deferred to main" and the merge re-escalates on main. Review caught two real gaps on the way: dead scripts left behind (deleted), and the claim that a Dependabot lockfile bump would get a full run after merge (it would not have, since a lockfile-only merge selects zero tests); fixed by escalating any merge that touches the lockfile to the full suite on main. Est. about $460/mo by the model above.
Red main, caused and fixed: the job-merge PR landed green, then main went red on a post-merge guard requiring a sensor tag on new drift tests (PR #7465 fixed it 13 minutes later; production promote held in between, then fired). The one-job fold also means a single ARM runner shutdown now takes out all three required names at once; reruns clear it.
Measure: re-pull the jobs API for 2026-09-10 and compare billed minutes by label against the 09-09 sample above.

Measured after the cuts (2026-09-10)

Comparing the two pulls straight up doesn't work. The 09-09 sample only covers 3,000 of that day's runs (the runs list caps at 1,000 per query); the 09-10 pull is the complete run list for its window. So the raw daily-total dollar figures from the two pulls aren't a fair before/after — the "after" total looks about 3x bigger mostly because it's counting far more runs, not because CI got more expensive. What's fair to compare is cost per PR push and per merge, since that cancels out how many PRs happened to run that day. Numbers below use the corrected per-minute prices (x64 $0.008, ARM $0.005, 8-core $0.014) and GitHub's real billing rule (every job rounds up to the next whole minute).

What	Before the cuts	After the cuts	Change
Cost of one PR's CI run (median)	$0.070	$0.055	-21% fewer jobs run per push (18 → 15, median), from moving the full suite off the PR path
Cost of one merge to main (median)	$0.498	$0.519	+4%, flat the ARM savings got eaten by more of that work landing on the pricier 8-core runner (32% → 41% of merge minutes)
Jobs running per PR push (median)	18	15	-17%
Share of all CI minutes on the old x64 machines	36%	12%	real, clean win this is the ARM move working as intended, and it doesn't depend on how many runs happened that day
Time from a PR push to CI finishing (median, real runs only)	21s	24s	flat
Time from a PR push to CI finishing (slow 10%)	419s	549s	up — the PR lane's remaining work (affected tests + the merged lint/type/build job) is now a bigger share of what's left
Main went red after a merge	8 of 28 pushes	58 of 104 pushes	looks much worse raw, but see below

On that last row — the red-main count needs unpacking, not just quoting. Grouping the 58 red pushes into distinct incidents (not counting the same broken thing twice while it's being fixed forward) gives 14 separate episodes, not 58. Together they left main red for about 9.2 hours total across the 15.3-hour window (the longest single stretch was 1.9 hours, right after the cut shipped) — that's the time production promotes were held.

In plain English: 9 episodes in the after window where main went red on tests the PR lane no longer runs; 5 episodes where the type-check/lint/build check passed on the PR but failed on main — a separate gap. The 9 are the direct, intended cost of PR #7461 — since the full 12-shard suite and the guard-pin checks no longer run on a PR, the first time they can fail is after the merge, by design (checked directly on 6 of the 9: each one's PR-time checks were green, and only the main-only full-suite/guard-pin checks failed). That's the tradeoff this page already named when the cut shipped, not a new problem. The other 5 turned out to have nothing to do with the code: in every one, GitHub shut the ARM runner down mid-way through the type check ("The runner has received a shutdown signal" in the job log), the step shows cancelled, and the folded checks job goes red. That is the known ~0.5% ARM runner kill rate, and two of this week's cuts made it hurt more: moving to ARM exposed the required check to it, and folding four checks into one job means one kill takes the whole check down. For comparison, the 09-09 sample had 3 red episodes holding main for 2.2 of 17.2 hours (13% of the time); the after window had 14 episodes holding main for 9.2 of 15.1 hours (61%). Merges kept landing at about seven an hour, so once one broke main everything behind it stayed red until the fix landed, and breaks overlapped: the 115-minute stretch was four broken test shards stacked on each other.

Decided 2026-09-10 (Fede: revert): the full-suite-only-on-main cut was reverted the same day (PR #7604, merged 17:05Z); PRs run the full suite at approval time again, the ARM move and the checks fold stay. Net estimated saving after the revert: roughly $100/month plus the ARM minute shift, labeled estimate. The runner-kill auto-rerun (option 1 below) is still open.

What to do about it (proposed, Fede decides): (1) auto-rerun a job when its only failure is a runner shutdown signal, cheap, no test tradeoff, removes the 5 runner-kill episodes; (2) run the full suite once at merge time through a merge queue instead of on every PR push, which catches the other 9 before main at a fraction of the old approval-time cost; or (3) roll back the full-suite-only-on-main cut and give back roughly $460/month of the estimated saving.

Decided later on 2026-09-09: full suite only on main

Fede: "we should not run all tests on PRs." The full 12-shard suite now runs only on pushes to main, on the nightly schedule, and by hand; never on a pull request or a review verdict. Affected tests still run on every PR push. Production is unchanged: the web promote and the Lambda deploys already wait for the full run on main, and a red main holds every deploy until the next green commit. Built as its own PR (est. about $460/mo by the model above). The earlier objection on this page (red-main risk) was checked against 09-07 to 09-09: none of the 36 red-main runs were full-suite failures.

What other high-velocity and agent-heavy teams do (web sources fetched 2026-09-09, dated 2025 to 2026): Google's engineering book says to run only fast, reliable tests before merge and accept some coverage loss there, with the full run and automatic culprit-finding after merge (abseil.io/resources/swe-book/html/ch23.html). Stripe runs about 5% of 1.2 million tests pre-merge, selected by file dependencies (stripe.dev/blog/selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorepo). Uber: affected tests pre-merge, full validation post-merge, at 2,500 changes a day (uber.com/blog/flaky-tests-overhaul). GitHub merge-queue guidance and Aviator: affected before, full after, bisect on batch failure rather than auto-revert (aviator.co/blog/pre-and-post-merge-tests-using-a-merge-queue). A 2026 Mergify report and an arXiv preprint claim AI-assisted PRs break main about half as often as human ones; vendor and preprint figures, not settled. Two things we are not doing that the sources push: a batching merge queue (we tried one for a week in August and retired it, ADR-0107) and measuring the miss rate, i.e. how often the main run catches what affected tests did not. The miss rate is the number to start tracking once this lands.

Cross-check 2026-09-09 against Gera's portfolio architecture design (docs.propflowai.co/a/portfolio-architecture-how): no conflict. It plans three more proof lanes in the portfolio harness workflow; new lanes should use the ARM labels and run affected tests on PRs, full only on main, so they do not undo the cuts above.

Decision needed from Fede (original ask)

1. Pick A, B or C. Recommendation: A. If A, the PR gets closed with a note and the Blacksmith app can stay installed at no cost for a later test.
2. Approve the mechanical cuts (arm-or-disarm fold, reporter folds, preview mint and remove fold) as one PR. Recommendation: yes.
3. Quality Gate POC: keep on every PR, gate behind a label, or retire. Recommendation: gate behind a label until someone names what it catches.
4. The two design decisions (async review executor; full suite on review verdicts) go to the six-week backlog with a named owner, or one is picked now. Recommendation: backlog, Gera named, since both touch the review pipeline he owns.

Method and caveats

The sample is 17.6 hours of one busy weekday, not a two-week average; the runs API caps listings at 1,000 and the full window holds about 98,000 runs. Treat monthly figures as order of magnitude.
Prices are list rates per label; the invoice will differ by storage, by the Team plan's included minutes on standard runners, and by any negotiated rate.
Blacksmith numbers are one or two runs each. Directions are reliable; magnitudes are not.
Side finding, unrelated: the renewal conversation scoping eval fails on main today (Clara implies a maintenance request was filed when it was not). It only fires when its own files change, so it has been red silently. Belongs to the renewals backlog.

PropFlow Docs