Why the AWS bill jumped 6× on Sep 13, why nobody heard about it until Sep 16, and the guardrails that make the next one a same-day Slack alert.
2026-09-16 · account 194925971150 · us-east-1 · written by Agent Smith · guardrail work: operator op-aws-spend-guardrails
One sentence. We turned on self-hosted CI runners (RunsOn) on Sep 13; AWS Config was set to write a billable note about every resource change in the account; the runners create and destroy ~1,500 resources a day, so Config wrote ~120,000 notes a day at $0.003 each — and every alarm that could have said so was pointed at a personal inbox, a filtered budget, or an instrument that reads $0 because credits net it out.
| Day | Config notes written | Config $ | Whole-account usage $ | |
|---|---|---|---|---|
| Sep 12 (before) | 1,726 | $5 | $33 | |
| Sep 13 — RunsOn stack created 02:51Z | 112,565 | $338 | $445 | |
| Sep 14 | 118,508 | $356 | $537 | |
| Sep 15 | 125,844 | $378 | $557 | |
| Sep 16 (partial, override applied 23:51 CT) | 52,200 | $157 | $220 |
Source: Cost Explorer GetCostAndUsage, DAILY, filtered RECORD_TYPE=Usage, UsageType ConfigurationItemRecorded. Config's $378 on Sep 15 matches the anomaly email's "$373.77 max daily impact". Total Sep 13–16 Config: $1,229; anomaly detection's figure at send time was $1,058.
PutConfigurationRecorder + StartConfigurationRecorder, CloudTrail), an hour before Security Hub was enabled. Security Hub needs Config; the "all types, continuous" default was the path of least resistance. At that point the account had a few hundred resources and Config cost ~$5/day. Harmless.runs-on CloudFormation stack was created (self-hosted GitHub Actions runners). Every CI job now creates an EC2 Fleet + Instance + network interface + disk, runs ~10 minutes, and terminates them. We measured 526 instance launches in a single hour on Sep 17.ResourceCompliance note per Security Hub rule evaluated against them. Config now tracks 61,288 resources — 35,107 EC2 Fleets, 5,004 network interfaces, 4,999 disks, 2,645 instances — all dead CI runners.Neither half was wrong on its own. Config-on-everything is a fine default for a quiet account. RunsOn is a fine CI choice. Nobody joined the two, because nothing in the account states "ephemeral fleets × per-change recording = per-change billing".
Lambda was $78 month-to-date, not the $2K the console bar suggested (that bar is the whole account). It did have its own small spike, unrelated to Config: propflow-appfolio-sync-prod went from ~14 compute-hours/day to 178h on Sep 11 and ~70h/day since, and propflow-stage-replicator-prod ran 250h on Sep 13. Anomaly impact $50 total. Tracked separately by the guardrail operator; not this RCA's cause.
| Alarm that existed | Where it pointed | Why it didn't help |
|---|---|---|
AWS Cost Anomaly Detection (Default-Services-Monitor) | One subscriber: a personal Gmail inbox. Frequency DAILY. | AWS itself needs 24–36h of billing data to call something an anomaly, then batches to a daily email. Threshold ≥$100 and ≥40%. Nothing to Slack, nothing to Smith. First human sighting: Sep 16. |
Budgets: runs-on-ci $150, runs-on-ci-monthly $150 | Email at 80% / 100% | Both are tag-filtered to RunsOn's own resources — Config spend is outside them by construction. They fired for the $200 of CI compute, which was the expected part. |
Budget: runs-on-app-daily-budget $5/day | SNS topic runs-on-AlertTopic | The topic has zero subscribers. It fires into nothing. (Actual: $72/day against a $5 limit.) |
Our spend dashboard (/admin/spend, cron spend-snapshot) | DynamoDB rows, read by the dashboard | Blind by construction. src/lib/domain/spend/providers/aws.ts calls Cost Explorer with no RECORD_TYPE filter, so credits net every day to ≈$0. Prod receipt: [propflow-prod] SPEND_SNAPSHOT/aws#2026-09-10 amountUsd = −3.8e−08. The dashboard said "$0" while $557/day burned. |
| Agent Smith (on-call) | #alerts | No program in the Construct watches AWS spend. #alerts has no cost source at all. Smith found out when Fede pasted the email. |
smith-cost-fix-20260916 in CloudTrail). Two gaps: EC2 Fleet — the biggest type, 57% of tracked resources — was not included; and DAILY notes cost $0.012 each (4× continuous), so a resource that lives ten minutes still bills one note. Whether it helped is read from Sep 17's note count, not assumed.op-aws-spend-guardrails owns the guardrails below, as separate small PRs.| # | Guardrail | Closes which hole | Where |
|---|---|---|---|
| A | Smith watches real AWS usage. New Construct program aws-spend-watch: every 6h, one Cost Explorer read (usage only, credits excluded), per service, trailing 8 days. Pages #alerts when any service's day > max($25, 3× its 7-day median) or the whole day > 2× median. Also reads AWS's own anomaly list and posts anything ≥$50 once. One live message edited in place; repeats get a reaction, not a new post. | No cost source in #alerts; Smith blind | agent-smith + config/automations.toml |
| B | Fix the dashboard. spend-snapshot records usage and credits as two figures, so a fully-credited account never reads $0. Regression test fails on the old code. | Netted-to-zero instrument | propflowai src/lib/domain/spend/providers/aws.ts |
| C | Point AWS's alarms at Slack. Add the propflow-alerts SNS topic (already forwards to Slack via propflow-sns-slack-forwarder-prod) as a subscriber to the anomaly subscription, frequency IMMEDIATE, threshold $50. Create one account-level monthly budget (not tag-filtered) with 50/80/100% alerts to the same topic. Give runs-on-AlertTopic a subscriber. | Wrong inbox · filtered budget · empty topic | AWS (additive, dry-run first) |
| D | Stop the bleed. Fede's pick from the decision above; receipt = Sep 18's note count back under ~2,000/day. | The cause itself | AWS Config recorder |
Done means: replaying the Sep 13–15 data through guardrail A produces a page (the detector must fire on the incident it exists for); the dashboard's AWS row shows real usage from prod; a test publish to propflow-alerts lands in Slack; Config notes back under ~2,000/day.
Once the audit-log accident was dead, the CI runners were the whole bill. This section is the takeover from Fede's session on 2026-09-18, source: Cost Explorer usage records and RunsOn's own job table (DynamoDB runs-on-workflow-jobs), day 2026-09-17.
| Sep 17, one full day | |
|---|---|
| Runner machines, full price (spot cap hit) | 151 h, $43.80 |
| Runner machines, spot | 178 h, $29.81 |
| Runner disks (provisioned 400 MiB/s) and public IPs | $8.48 |
| Machines launched | 7,323 |
| Full-suite runs (12 shards each) | 423, of which 339 on PR branches (245 push-triggered, 95 verdict) |
| Full-suite shards | 5,029 = 73% of paid job minutes |
| Median job | 2.0 min; boot + teardown ≈ 79 of 329 machine-hours |
| Latency: full suite / affected lane (median, p90) | 5.0 / 13.7 min; 2.0 / 3.0 min |
Not the circuit breaker. RunsOn's log states it: Got MaxSpotInstanceCountExceeded error. You need to adjust your spot quotas with AWS. Zero breaker events, zero fleet API errors on Sep 17. The spot quota was 256 vCPU (32 machines); peak concurrency was 60. A quota increase to 512 vCPU was requested on 2026-09-18 (AWS case 178974869700597), pending.
| Lever | Expected effect (estimate) | Side effect | PR | Status |
|---|---|---|---|---|
| Spot quota 32 → 64 machines | ~$19/day | none | (AWS quota request) | pending with AWS |
| Full suite once per review round on hub-sized PRs, not on every push | removes ~245 of 342 daily branch full suites | big PRs merge-ready ~5 min later (median) | #9480 | open, auto-merge armed |
| 4 shards instead of 12 | 4 boots per full suite instead of 12 | full-suite wall clock a few minutes longer | #9479 | open, auto-merge armed |
| Runner disks at gp3 baseline throughput | ~$4–5/day | none expected | #9482 | open, auto-merge armed |
| Cancel superseded runs | ~$3/day | unsafe: push and verdict runs are kept in separate groups after four incidents | — | rejected |
Labeled estimate: runners from about $82/day ($2,460/month) to $20–25/day ($600–750/month). Proof: the same two queries (RunsOn job table: launches, full-suite runs on branches, median job minutes, full-suite wall clock; Cost Explorer: BoxUsage vs SpotUsage hours and $, EBS:VolumeP-Throughput.gp3) for the first full UTC day after the three PRs merge, added here as an "after" column. Not yet measured.
The account's non-CI baseline was $14.69/day on Sep 1–12 (about $440/month). The $300–500/month target is reachable only by moving the three heavy jobs back to GitHub-hosted runners. GitHub's own Actions usage for the org was $4,290 gross on Sep 1–18, 100% discounted; the discount's size and expiry are being looked up. Decision pending Fede.
CreateStack runs-on 2026-09-13T02:51Z · PutConfigurationRecorder/StartConfigurationRecorder 2026-08-15T02:24Z · EnableSecurityHub 2026-08-15T03:34Z · PutConfigurationRecorder (daily override) 2026-09-17T04:51Z.aws configservice describe-configuration-recorders: allSupported=true, CONTINUOUS, recordingScope PAID; overrides DAILY for Instance/NetworkInterface/Volume.aws configservice get-discovered-resource-counts: 61,288 total; EC2Fleet 35,107.aws ce get-anomalies: Config $1,057.85 (max $373.77/day) · EC2 $200.37 · Security Hub $19.41 · Lambda $50.58.aws ce get-anomaly-subscriptions: one subscriber (personal email), DAILY, ≥$100 AND ≥40%. aws budgets describe-budgets: three runs-on-scoped budgets, none account-level. aws sns list-subscriptions-by-topic runs-on-AlertTopic: empty.smith-db query SPEND_SNAPSHOT --prefix aws#2026-09-1 → [propflow-prod] amountUsd ≈ −3.8e−08 per day.