Proposed — pending reviewInvestigation + decision
Do we need to run every test like this?
CI cost forensics, 2026-08-03 · every number below is measured from the GitHub jobs API and the org billing usage report unless marked dotted = estimate · v2 — corrected after the deeper forensics passes completed
Correction (v2, later on 2026-08-03) — the deeper passes overturned v1's central premise
v1 claimed the affected-tests path was dead and ~89% of runs escalated to the full 12-shard matrix. That was a measurement artifact: the full-shard jobs exist as skipped jobs in every run, and the v1 sample counted them without filtering by conclusion (inferred). Re-measured today with skipped jobs excluded: 0 of 40 sampled PR-event runs executed any full-shard job — every one ran the affected lane. The full matrix runs only on push:main, and only when escalation triggers (11 of all 25 main pushes today). The selector is alive and doing its job. Consequences: v1's Model A (−28%) and Model B (−37%) were computed from a false baseline and are withdrawn; the corrected arithmetic below shows hosted-side levers max out around −13%, which makes the self-hosted runner (Q2) the only decision that materially moves the bill.
What the investigation found
1 — Corrected: the affected-tests path is alive. CI is already two-tier.
Measured today, skipped jobs excluded: 0 of 40 sampled PR-event runs executed a full-shard job; all 40 ran the affected lane (avg 11.2 billed min/run). The full 12-shard matrix fires only on push:main when escalation triggers — 11 of 25 main pushes today, avg 81.8 billed min vs 13.9 for non-escalated. And since 2026-08-02 the expensive lane on PRs is gated behind the reviewer's verdict, not every push. The two-tier model this doc proposed substantially already exists; what v1 read as "89% full escalation" was skipped-job counting.
2 — Corrected: the bill is per-push overhead, not the test matrix.
CI's 3,742 min/day decomposes as: ~2,644 min of PR-event runs (≈254/day × ~11 min — and the tests inside are only ~4 min of that; the rest is lint, type check, build, DDB gate, await-Vercel-preview on every push), ~750 min of full-suite premium on main (11 escalated runs), and ~350 min of non-escalated main overhead. Two measured waste pockets: 22.1% of PR runs are superseded by a newer push (n=1,182 over 4.66 days) — cancel-in-progress already exists (ci.yml:101), but cancels land late: superseded runs still burn ~70% of a normal run's minutes, ≈436 min/day wasted. A "push debounce" would re-implement what's already armed; the real lever is an early cheap "still-head?" guard step before each job's expensive setup.
3 — Everything else is already trimmed.
After correcting a measurement artifact (API-written check-runs with hours of wall-clock and zero runner time, which briefly made "Dependabot Review Gate" look like 20% of the bill — its real cost is ~4 min/day), no non-CI workflow exceeds ~5% of the day. The July cost program (10 PRs) did its job on the long tail. The bill is now dominated by fixed per-push overhead across ~280 runs/day — a cost that scales with push volume, not test strategy, which is why hosted-side levers cap out low and hardware (Q2) is the decision that matters.
| Workflow (today, corrected) | Billed min | Share |
|---|---|---|
| CI — of which unit-test shards 2,374 + aggregator 275 | 3,742 | 54% |
| Claude Code | 375 | 5.4% |
| Deploy Lambdas | 341 | 4.9% |
| Semgrep SAST | 319 | 4.6% |
| Claude Code Review | 311 | 4.5% |
| Everything else (~20 workflows) | ~1,800 | ~26% |
The philosophy
The test suite is more load-bearing here than at a normal company — agents write the code, a bot reviews it, and auto-merge ships it. Nothing about this proposal runs fewer tests before code reaches a tenant. The claim is narrower: the safety invariant is "the full suite has passed on the exact tree that ships, before it ships" — and that requires one full run per merge candidate (or per deploy batch), not one per push. Everything before the final state only needs directional signal: type check, lint, and an affected slice, in single-digit minutes.
What remains to do — corrected proposal
The two-tier structure v1 proposed already exists: PRs run type check + lint + build + DDB gate + affected tests; the full matrix proves main. The remaining hosted-side levers, in order of return-per-surgery:
Lever 1 — late-cancel guard (small PR): an early "am I still the head SHA?" step in each job, before the expensive setup. Recovers most of the ≈436 min/day that superseded runs burn after cancel-in-progress has already fired but before the cancel lands.
Lever 2 — batch the full suite (merge queue / deploy-batch): today the full matrix runs on 11 of 25 main pushes (~68 min premium each, ~750 min/day). Batching main proofs to ~3/day keeps the invariant — the full suite has passed on a tree containing everything that ships, before it ships — for ~550 fewer min/day, at the cost of required-checks surgery and bisect-on-red-batch.
Lever 3 — per-push overhead diet (unmeasured): the largest slice (~2,200 min/day) is lint/typecheck/build/DDB-gate/await-preview repeated ~254×/day. Whether any of it can defer to the verdict-gated lane is a follow-up measurement, not a decision today.
Current vs projected — the corrected arithmetic
Inputs, measured today unless dotted: CI = 3,742 min/day (PR events ≈2,644 + push:main 1,094); late-cancel waste ≈436 min/day; full-suite premium on main = ~750 min/day (11 runs × ~68 min, fully measured); blended price $0.00566/min. v1's Models A/B are withdrawn (computed from the false 89%-escalation baseline).
| Model | CI min/day | Total min/day | $/day | $/mo | vs today |
|---|---|---|---|---|---|
| Today (measured) | 3,742 | 6,900 | $39.88 | $1,196 | — |
| A′ — Late-cancel guard Recovers ≈350 of the 436 superseded-run min/day. One small PR, no required-checks changes. | ~3,390 | ~6,550 | ~$37.90 | ~$1,137 | −5% |
| B′ — A′ + batch full suite on main Full matrix ~3×/day instead of 11 (≈550 min/day saved). Merge-queue / deploy-batch surgery on required checks. | ~2,840 | ~6,000 | ~$34.80 | ~$1,044 | −13% |
| C — self-hosted runner Full runs, Semgrep, deploy workers and LLM-wait workflows on owned hardware at $0/min. Meter keeps only cheap one-offs ≈ 1,100–1,400 min | ~450 | ~1,250 | ~$7.10 | ~$213 | −82% |
The honest summary, v2: CI is already smart — the hosted-side levers left are worth −5% to −13%, not v1's −37%. If the 50%-cut goal (let alone −80%) is real, there is exactly one lever that reaches it: the self-hosted runner. It was approved (#4987) and reverted (#5015) the same afternoon as a policy call — "no self-hosted runners" — on machinery that was never provisioned; nothing broke. The full story and re-adoption blockers are under Q2.
What this does not change
Coverage. Every test that runs today still runs against every tree that ships. Merge-time blame isolation is preserved in model A/B (one candidate per full run). Deploy gating (#4924) is untouched. A red full run still blocks exactly what it blocks today.
Risks, stated plainly
① The affected-selector has missed before (the types.ts hub-file incident) — that risk exists today, and the main-push escalation logic is the backstop; batching full runs (lever 2) widens the window between a selector miss and its catch from one main push to one batch. ② A red batched run needs bisect-and-revert and blocks deploys meanwhile; flakes hurt more when one run vouches for several merges. ③ Batching changes the required-checks wiring — review, Unit Tests Result and the Dependabot gate all need re-pointing, exactly the kind of change the .github fence forces a human to merge. ④ The cancel guard (lever 1) has no coverage risk — it only skips work for SHAs that are already superseded.
Decisions
Pick one per question — answers persist and your session reads them back.
Q1 — Which hosted-side levers to ship? (v2: two-tier itself already exists — this is what's left)
Q2 — Bring back the self-hosted runner (the −80% ingredient)?
The revert story (measured from the repo record, 2026-07-30): #4987 landed at 16:20 UTC and the revert PR #5015 opened five minutes later; total lifetime on main was 1h54m. Nothing broke and nothing regressed — the runner was never provisioned (the Mac mini was LAN-unreachable at cutover; verified today: zero registered runners, the routing variable never set), so every workflow ran hosted the whole time. The stated reason, verbatim from #5015: "Fede's decision: no self-hosted runners… there is no reason to carry unused routing machinery, a canary for a runner that will not exist, and a provisioning script nobody will run." So Q2 is not "is the blocker fixed" — it is whether that call gets revisited now that it is the difference between −37% and −82%.
If revisited, the real blockers (all flagged in #4987's bot review, none fixed since): the mini's reachability/sleep behavior (LaunchAgent loads at login, not boot — a sleeping mini wedges the required check); a macOS toolbox gap (envsubst missing → exit 127, silent jq degradation); an alerting inversion where the likeliest cutover typo hangs all merges without paging; and canary trust (#5027 still open). The bot review noted twice that a Linux host eliminates the toolbox gap entirely and most of the availability risk. The design itself is recoverable with git revert 5007d2339.
Q3 — Tier the suite itself (core invariants gate shipping; long tail runs nightly)?
Notes
Sources (v2): GitHub org billing usage report (per-day gross/minutes); jobs API — 1,500-run supersession window 2026-07-30→08-03, 146-run job-level waste sample, 40-run PR full-vs-affected re-verification with skipped jobs excluded, and all 25 of today's push:main runs measured individually; ci.yml at HEAD for the concurrency and verdict-gated lane config; #4987/#5015 PR record + live runner-registry check for the Q2 revert story. All forensics passes are complete; v1's 45-run sample (the withdrawn 89% claim) is superseded by the skipped-job-filtered re-measurement.