CI & Infrastructure · 2026-07-30

GitHub Actions cost program — what shipped, what it saves, what's left

Overnight sprint, Jul 29→30. Every number below is from the GitHub API (run census with 1-minute-per-job billing rounding); figures marked est. are extrapolations from sampled job durations.

9 PRsmerged (7 overnight + 2 this morning)
~3,700 min/dayest. removed at Jul-29 volume (~$20/day)
~23%of the 16,559-min worst day; ~35–45% of a normal ~6.6k day (est.)
Both declinedthe two levers past 50% were reviewed and rejected

Why the bill exploded

The Jul-29 bill ($89.62, 327% of July's budget pace) was volume, not regression — runs tripled Jul 27→29 with the fleet push, and all three earlier cost rounds were still active. But the census exposed a structure worth attacking: 38% of the worst day was billing-rounding tax (jobs that do seconds of work billed a 1-minute minimum), and another ~1,500 min/day was runners idling on purpose — polling loops waiting for other systems.

What shipped (all merged, all behavior-verified on live runs)

ChangeBefore → after (measured)Est. daily effect
Deploys fire off CI completion instead of polling for itgate job 10–12 min of billed waiting → 79 s, zero await steps; no-op deploys skip free~700–1,000 min
Auto-merge evaluator ignores events that can't change its decisionevery bot-review event billed 1 min → 21 of 23 now skip at $0~600 min (worst day)
Docs-lane CI collapsed to one job (checks posted via API)4 jobs/run → 1 (4.0 → 1.0 min/run)~450 min
Nested package CI collapsed to one gate job6 jobs/run → 1 (2.3 → 1.0 min/run)~480 min
Two PR gates skip free instead of booting a runner to say "nothing to do"always-billed → 64% of override runs now $0; eval gate runs only on prompt paths~740 min
Voice sim no longer triggers on app code it provably can't observe25 of 53 sampled runs were unobservable triggers~250 min
Vercel-preview await skips free unless a preview was opted intobilled on all 257 PR events (Jul 29) → $0 on the ~95% common path (merged, skip verified live)~257 min
Lambda bundle check → arm64 + node_modules cachex64 + full npm ci (~3 min/run) → ~15 s cache restore~150–200 min

The incident the rollout caught (and why it was worth it)

The first real deploy after the trigger conversion failed all six lambda promote lanes with a false "version skew": a latent guard bug (a git command that echoes a ref name instead of failing empty) that the old trigger had always masked. Production was never at risk — the guard sits before the alias flip, so the previous versions kept serving — but deploys were wedged for ~5 hours overnight. The fix merged the same night, review caught two real holes in the first cut (a fast path that would have blessed a feature-branch manual deploy, and a slow path that could warn but never fail), and the pipeline then proved itself: all six lanes promoted green, and the next two organic deploys succeeded end to end.

Honest distance to the 50% goal

Shipped changes remove an estimated ~3,700 min/day at Jul-29 volume — not 50%. What remains is dominated by two things only a human decision unlocks:

Decision 1 — Self-hosted runner pool for the LLM-wait workflows (code review idles ~650 min/day est. waiting on model responses) and optionally the CI shards (~7.7k billed min/day). Biggest single lever, roughly $5–6+/day at current volume. Trade: runner ops burden and queue management.

Decision 2 — ADR-0120 (Proposed — pending review): coalesce hub-escalation full-suite runs. During fleet-push storms, 3–5 overlapping full suites each test a tree the next one strictly contains; a newest-wins deferral with an "owed-full" marker recovers an estimated 800–1,000 min/day on heavy days. Measured basis: all 42 inspected Jul-29 escalations were hub-sized, zero were red-main. Trade: a red full suite covers a burst of merges instead of one (bisect or the next per-push escalations localize it). Per-diff lanes keep their no-coalescing guarantee untouched.

Status: the ADR is a proposal held for review — nothing in it is live. The full decision text, rejected alternatives, and rehearsal plan are in the repo's ADR index (0120) and will publish here automatically when merged.

Final ledger — where this landed

BucketEst. min/dayStatus
Shipped (9 PRs)~3,700merged, behavior-verified on live runs
ADR-0120 coalescing~800–1,000 (heavy days)rejected — blame granularity on a red burst valued above the recovery
Self-hosted runner pool~$5–6+/dayrejected — built, then reverted; nothing self-hosted exists

Final position: est. ~$21/day saved (~45% of a normal day, ~23% of a storm day) — short of the 50% target. Both levers that would have closed the gap were reviewed and declined, so the target stands unmet by choice rather than for want of options. Remaining unexploited levers are small (the test-aggregate job floor, ~$1.70/day). All figures are estimates from sampled job durations; the daily spend reports are the real proof.

Smaller levers, documented and rejected/deferred

Test-aggregate job floor (~318 min/day — touches a required check, needs care) · Semgrep fold rejected (its pinned container is the reproducibility contract; not worth ~$1/day) · a dormant sandbox deploy lane (metric-snapshot preview-real) fails its own smoke and deserves a look someday.

PropFlow Docs