AWS bill jump, October 2026: what drove it

"CI-tagged" means every resource the runner stacks label Purpose=ci: machines, disks, IP addresses, logs, and the runner control plane. "Not CI" is DynamoDB, Lambda, GuardDuty, Config, ECS and similar. It runs $29–57 a day and has not grown. The Sep 13–17 spike was AWS Config recording every runner, already fixed (RCA). Sep 13–21 was a first RunsOn trial at $33–98 a day of EC2. It was off Sep 22–28.

Credits: $309 used in Aug, $4,012 in Sep, $678 in Oct, about $5,000 in total. Credits covered $99–415 a day from Oct 1–3, then only $5–25 a day from Oct 4–6. That points to the pool being spent (an inference from the credit line, not from a balance page).

Size buckets and the full-price column leave out instance-less lines such as CPU credits, so they read a little lower than the totals (full price incl. those: $370 on Oct 2, $51 on Oct 6). Everything is arm except a few x86 2-cpu spot machines. Three runner fleets: the main us-east-1 stack, a 6-zone "flex" stack (since Oct 1) and us-west-2 (since Oct 1). On Oct 6 full-price machines cost $31 in the main stack, $10 in flex and $9 in west. Spot ran $157.

Did the machine count or size go up? Yes, at the queue launch. Work per merged PR went from ~25 vCPU-hours to 63 on Oct 2. It is back to 33 as of Oct 6. The thin queue (Oct 4) moved the 64-core full suite out of most batches. Price per vCPU-hour fell to about $0.02 as more ran on spot. PR sharding (#16955, merged Oct 6 10:42 UTC) has not shown up as a cost rise yet: the 4-cpu bucket, where PR unit tests run, fell from $76 to $30 between Oct 5 and Oct 6. Oct 7 is its first full day.

What CI costs besides machines (Oct 6, about $38): public IPv4 addresses $10 (about 100 runners live at once, half a cent per hour each), disks $9, CloudWatch log ingestion and vended logs about $8, data out $7, runner control plane (API Gateway, DynamoDB, SQS) about $7.

Merged PRs are counted by UTC day from GitHub search. The GitHub-era row comes from the CI cost study.

• Full-price machines: none of the workflows on main still asks for a full-price machine. What was left on Oct 6 ($51) was the always-on 2-cpu glue pool in flex (switched off by #17223 at 21:53 UTC Oct 6) plus spot fallback when AWS was short of capacity. Gera approved a looser fallback switch (20/15/10 to 40/15/5) on the bridge thread, and CICD applies it. #16907's queue-checks pin was reversed by #16996, and #16992 moved the last pinned jobs to spot (both Oct 6 morning). At 06:27 UTC Oct 7, one full-price machine was running out of 84.
• Idle or orphaned: none found. The oldest running runner was 52 minutes old. There are 3 stopped full-price t4g.medium in flex, probably left from the old pool, which bill disk only. There is 1 unattached disk and 0 unused IP addresses.
• Correction to the bridge thread: the "$7.5 NAT data processing" line is CloudWatch log ingestion (Cost Explorer files it under CloudWatch). Neither region has a NAT gateway. So moving runners off public IPs would need a new NAT gateway: a fixed fee plus a per-GB charge. Gera's hold on that change, pending a GB-per-day estimate, still fits.
• Release skip (#17236): its effect can't be split out in Cost Explorer yet. On the bridge thread, it skipped the recheck on 7 of 10 overnight releases. The daytime re-measure decides it.

Factory Cost tab, queue machine time at spot price, last 7 days (about $114/day): CI Checks $45/day (863 jobs, 3,220 vCPU-h), Guard pins $40/day (4,612 jobs, 2,890 vCPU-h), Unit Tests (affected) $9, full set on one machine $8, full set $6.5, fail-fast $3. 51.5% of queue machine time is waste, about $59/day.

1. Queue waste ($59/day, measured). Two parts: re-runs behind a red PR, and spot machines lost mid-run. #17231 (retry a reclaimed shard inside the run) and the false-alarm fixes landed Oct 6–7. Replay-before-queue is approved. Fewer concurrent batches would also shrink the blast radius of one red. Estimate $30–45/day once these settle. Risk: fewer batches means lower merge throughput.
2. Full-price residue (up to about $40/day). Mostly removed already, see section 4. Don't count the saving until Oct 7–8 show full-price EC2 under $10/day.
3. Queue CI Checks on 16 cores ($45/day). That is about 2.9 runs per merge at about 3.7 vCPU-hours each. 8 cores would save $10–20/day if the job less than doubles in time. Risk: a few minutes longer per batch, which pushes against the one-hour open-to-live goal.
4. Guard pins (4,612 jobs/day, about 15 per merge). Already halved to 3 shards of 8 cores in the queue. The one-machine shadow v2 is measuring the "share one machine" idea, and it expires Oct 8. Decide after its numbers come in, not before.
5. Small: public IPv4 $10/day (on hold), CloudWatch logs about $8/day (shorter retention or less verbose runner logs), three runner control planes about $7/day (fold back to one once flex is the only fleet).

AWS Cost Explorer daily unblended cost, record type Usage (credits and tax excluded unless named), grouped by service, usage type, instance type, purchase type, and the stack and Purpose cost-allocation tags. EC2 describe calls for live instances, volumes, addresses, NAT gateways and endpoints in us-east-1 and us-west-2, read-only, with nothing changed. GitHub billing usage API (actions, gross). GitHub search for merged PRs. The factory Cost tab for per-job queue figures. The engineering bridge thread (Oct 6–7) for what CICD already owns. Per-workflow AWS cost is not available, because the runner tags runs-on-workflow-name / runs-on-workflow-job-name are not activated as cost-allocation tags. Turning them on would give exact per-job dollars, but it is an account setting, so that decision is left to Fede/Gera.