AWS cost incident, 13–17 September 2026

What ran up the AWS bill, why nobody noticed for four days, and the specific guardrails that would have caught it. Written for Fede, answering the five questions he asked in #agent-smith.

2026-09-17 · AWS account 194925971150, us-east-1 · every figure read live from the Cost Explorer, Config, Budgets, SNS and CloudTrail APIs on 2026-09-17 · Sep 16 and Sep 17 figures are still settling and are marked *

The short version

Independent audit — 2026-09-17 afternoon

Second pass by a separate session, asked by Fede to check Smith's and Trinity's work, review this RCA, and audit everything else still costing money or close to timing out. Every number below was read from AWS, GitHub and Slack on 2026-09-17 between 14:00 and 17:00 UTC. Sep 17 is not in Cost Explorer yet, so anything labelled "pace" is a projection from Sep 13–16 actuals.

1. What is actually true right now

One scary-looking finding that is false. CloudTrail shows 28 code updates to the Slack-forwarder Lambda overnight under the fede-admin key from cloud hosts. Those are GitHub-hosted CI runners running the deploy workflow on every merge, which redeploys every Lambda whether or not it changed. Not a hand edit. It is wasteful (a dozen redeploys per merge) but it is the sanctioned CI/CD path.

2. Projected spend now

3. What is still burning, and the fix for each

3a. CI runners: 6× the plan, and it is the mix, not the price

The RunsOn design (2026-09-13) priced the fleet at ~$300/mo on spot. Compared with what happened:

Why so much on-demand. Not the circuit breaker: its circuit_breaker_active flag was false in all 49 interruption events sampled, and only 1 of 570 spot requests failed for capacity. The wall is the account quota: 256 vCPU for spot and 256 for on-demand, each worth 32 of these 8-core runners. At peak (34 runners launched in one 30-second window) the spot pool is full, RunsOn falls over to on-demand, and on-demand then hits its own cap (VcpuLimitExceeded in the RunsOn logs). Volume is the other half: ~1,877 runs/day × ~14 jobs/run, and 53% of runs are triggered by review events (deliberate, see the workflow header, but it is the single biggest volume lever).

Levers, all estimates: raise the spot vCPU quota to 512 or more (free, AWS form) ~$500–700/mo; broaden the instance families beyond c7g so spot has more pools ~$50–150/mo; fewer shards per run (14 jobs today) ~$200–300/mo; stop rerunning the full lane on review events, or run only the cheap checks then, up to half the volume; re-add the S3 build cache once per-ref isolation is verified ~$20–40/mo. Even with all of them, at today's CI volume the fleet lands above $300/mo. The plan underpriced the volume, not the machine.

3b. DynamoDB: one org-wide query, run once per property per tick

The household shell sweep inside the AppFolio sync calls listHouseholds(organizationId), an org-wide read on entityType-createdAt-index, once per property per tick, then filters to one property in memory. Western Slope's portfolio went from 2–15 in-scope properties to 71 around Sep 12, so the same query now runs ~44× per tick per account, every 1–5 minutes, around the clock. That is the flat ~640k reads/hour. The phone index climbs with it (one lookup per candidate household). Reads on that index Sep 3–16: 69.5M. Fix: read the org's households once per run and hand the list to each property. Dark PR propflowai #9104 merged 15:51 UTC. It did not reach the Lambda until 16:29 UTC: the deploy workflow builds whichever commit's main CI just finished, and that commit's CI went red on a test-runner timeout (all 7,761 tests passed), so two later deploys shipped older code. Rerunning the failed job fixed it. Expected effect: ~44 org-wide queries per tick → 1, roughly $150–200/mo. Before → after (household index, reads per 5 minutes): 43k–52k before the 16:29 UTC deploy → 18k in the first window, then 4k–11k. Roughly 85–90% fewer reads on that index, from ~600k/hour to ~80k/hour.

3b-ii. DynamoDB writes: the same prospect rows rewritten every minute

Found while taking over Smith and Gera's bridge question. Table writes went from ~0.5M units/day (Sep 12) to ~3.35M/day (Sep 13 onward). Read one tick apart in prod: about 64 prospect rows fleet-wide are rewritten every minute with only updatedAt and version changing (version counters 5,000–69,000; Camellia 23 of 25 rows, Western Slope 26 of 153). Each rewrite replicates to 4 indexes.

First guess was wrong. I attributed it to the household step re-saving members (PR #9122, merged and live 17:25 UTC). Its counters show it was not the writer; the guard is a valid cleanup, not the fix. The rows kept moving.

Confirmed cause (rental-application writer, ~39 rewrites/min): for approved or lease-signed rows whose stored approvedAt is missing, the writer sets it to "now" every tick, its change check compares it, and the DynamoDB projection allowlist never saves it. Rewritten every minute, never self-heals. Fix: persist approvedAt through the projection (dark PR, one concern).

Guest-card writer (~25 rewrites/min): its change check also fails on rows that look identical, but the failing field cannot be named from the stored data. Next deploy adds a churnFields= field-name count to the per-property summary line so production names it, then the fix follows. The guest-card fingerprint bug from Gera's session is real but small: two AppFolio guest cards mapped onto one prospect on ~1 in 10 churning rows; separate PR after.

Update 18:45 UTC — approvedAt fix live (PR #9134, deployed 18:22 UTC): rental-application rewrites at Camellia went from 28 per minute to 16–17; one row that had been rewritten 6,000 times stopped. The field-name counter (PR #9136) then named the rest, fleet-wide over 10 minutes: source flipped by both writers (52 + 49 times), af.coApplications and leadFacts judged different because of key order (24 + 9), af.unitId written by both writers (4 per tick per property), and a two-guest-cards-on-one-prospect case (17). Fixes in flight: structural compare for the order-only fields (auto-merge); "once an application is linked the guest card cedes source and unit to it" (a change in which lead source wins, waiting on Fede's yes).

3b-iii. The bigger writer: the per-property job lock, put and deleted 700 times a minute

Counting DynamoDB requests by operation (not by write units) showed the prospect loop was never the majority. Per 5 minutes the table takes about 2,500 creates, 1,300 updates and 2,000 deletes. Nothing in the product deletes 400 rows a minute. The deletes are the sync job lock added on Sep 10 (PR #7582) to stop two overlapping sync runs from creating duplicate prospects: before each job on each property the Lambda writes a lock row, and after the job it deletes it. That is one create plus one delete per property, per job, per minute, and the 1-minute schedule runs three jobs plus the household step: 87 properties × 4 × 2 ≈ 700 writes a minute, all pure overhead. It became visible on Sep 13 at 14:00 UTC, the hour Western Slope's 87 properties came online (deletes went from ~300/hour to 21,000/hour and stayed there).

Where the write units go now, per 5 minutes: table ~7,700 (locks ~4,200 of it, 55%); indexes ~7,100, driven mostly by the prospect rewrites because each rewrite is copied into 4 indexes. The lock rows also feed the table's change stream into two Lambdas (stage replicator, org drift detector) 40,000 times an hour, and the replicator logs an error for every lock row it does not recognise: ~560,000 error lines a day.

Decisions (engineering, no customer effect): one lock per property per tick instead of one per job and one for the household step (same protection, 4× fewer lock writes; dark PR); the replicator learns to skip lock rows quietly (one-line PR). Left for the replicator's owner: HOUSEHOLD rows are also unrecognised by the replicator today (~14 per minute), so households are not reaching the stage table — flagged on the bridge.

Result, 19:49 UTC — all four fixes live. The lock change (PR #9146) reached the Lambda at 19:38 UTC; the writer fixes (#9144 lead-source/unit ownership with Fede's yes, #9145 structural compare) at 19:17 UTC; the replicator quiet-skip (#9142) at 18:59 UTC. Per 5 minutes, before (18:39–19:14 UTC) → after (19:39–19:49 UTC):

Projected: table plus index write units ~3.8M/day → ~1.9M/day. What remains is named by the counter: two AppFolio guest cards resolving to one prospect (~44 rewrites per 10 min, issue #9150 with a proposed rule; it is a prospect-identity decision), and small tails on lead facts, inquiry id and application-received timestamps. Locks still cost one create and one delete per property per tick; the next step down, if ever wanted, is one lock per account per tick.

3c. AppFolio sync: 5× slower, from a leftover 1-second sleep

PR #7723 (Sep 10) merged the per-connection sync workers into one worker per AppFolio account, which walks every property in series with a 1-second pacing floor to protect AppFolio's rate limit. Western Slope's 71 properties now take 40–85 s per run instead of 8 s; the slowest runs hit 650–750 s against a 900 s timeout. Zero errors, zero throttles, just long. Correction after Fede asked "wasn't it supposed to be per org": the fetch is per account (one request per report per account since #7723), but the 1-second pacing sleep from the older per-property design (#7457) was left in and still fires once per property, including the ~65 properties that make no AppFolio call at all. That is ~70 s of pure waiting per Western Slope run, ~3,200 runs a day. Fix: sleep only when a property actually makes a live AppFolio request. Dark PR #9114 (152 lines, red-before/green-after test, new spacingSleeps= log field), auto-merge armed. Rate-limit protection is unchanged for the properties that still call out. No concurrency change needed.

3d. Stage replicator: 5× the invocations, 20× the logs

propflow-stage-replicator-prod went from ~45–120k invocations/day to 330–650k/day on Sep 13, with 95–239 errors/day on Sep 11–13, and now writes ~1.15 GB of logs a day into a log group that never expires. Some invocations hang to the full 900 s timeout while the p95 is 7.5 s. Not root-caused in this audit; it tracks prod write volume plus the DynamoDB churn above. Needs its own look (Gera's lane).

4. Performance and timeout risks (read-only sweep of every Lambda, table, queue, schedule)

Clean: no Lambda throttles in 14 days, no table scans in prod, every EventBridge schedule and every dead-letter queue empty, no NAT gateways or databases to tune, no Lambda timeouts on the customer voice path.

5. Alerts: what was ignored, and where they go now

• Personal Gmail (60 days): the only unacknowledged real alert was this one, the AWS Config anomaly, emailed Sep 14, 15 and 16 as it climbed $355 → $784 → $1,058. Twilio's A2P rejection was handled and approved. Nothing from Vercel, Sentry, GitHub or Temporal reaches that inbox; those go to the work address, which this audit could not read.
• Why it was in the Gmail: the AWS account was created with it as the root email, and AWS's default anomaly subscription emails the root address daily. That subscription is now deleted; the immediate Slack one (≥ $10) remains. Billing and security notices from AWS still go to the root email until it is changed in the console.
• AWS side now: every alarm, the runner budget and the anomaly monitor publish to propflow-alerts → #alerts. Still missing: an account-wide daily budget (all three budgets are tag-filtered to CI, so the Config spend was invisible to them).

6. Feedback on the incident response (Smith and Trinity)

What was good. The mechanism was found within 15 minutes and explained plainly. The credits-hide-spend trap and the $0 spend page were real catches. The topic-policy gap (repointing the budget would have silently delivered nothing) was caught before it bit. Smith checked for a newer decision before restoring his own overwritten change. Both bots said what they were not claiming.

What to fix next time.

• Two RCA pages for one incident (this page and rca-2026-09-16-aws-config-runs-on-spend), written by two sessions that did not see each other. One document per initiative. Both were also left describing a state that had already changed.
• Two sessions asked Gera the same decision 80 seconds apart with different options, and one applied a change the other overwrote four minutes later. A decision gets one owner and one card.
• "Done" claims outran the evidence: the daily cost check was reported as watching AWS "itself, merged and live" while it is paused and has never run; the alert times were an hour off; the first fix skipped the biggest resource type after naming it in the first message.
• The projection was wrong by a third ($2,000 vs ~$3,150/mo) because it stopped at the incident's own line. Once the loud item is fixed, re-baseline the whole bill.
• The $5/day "notes" number confused Fede. That is the pre-incident Config baseline for the whole account, not the runners. Say what a number is the price of.
• Two open PRs (#586, #588) were left as loose ends after the thread went quiet. Merge or close before calling the lane done.

7. Decisions for Fede (recommendation first)

All are AWS production settings, so each needs a yes. None touches a customer-facing behaviour.

1. Turn deletion protection on for every prod DynamoDB table. done 16:30 UTC Fede said yes via the orchestrator session. All ten propflow tables now protected (agent-jobs, claude-account-leases, status-cache, traffic added; prod, auth-prod, consent, dev, stage, auth-dev were already on). Left off on purpose: auth-stage, smoke-results, the two runs-on tables.
2. Set 60-day retention on all 26 unbounded log groups and a lifecycle on the CloudTrail and Config buckets (cold at 90 days, expire at 400). Recommend yes.
3. Raise the three queue visibility timeouts (inbound to 1800 s, agent-jobs to 3600 s). Recommend yes.
4. Create one account-wide daily budget at $60 that alerts to #alerts. done 16:30 UTC Budget account-daily-guardrail: $60/day, usage not net of credits, no tag filter, alerts at 100% to propflow-alerts. AWS already shows it in ALARM ($523 counted against today), so the first message to #alerts should arrive on the next Budgets evaluation cycle.
5. Request the spot vCPU quota increase (256 → 512) so the runners stay on spot. Free. Recommend yes.
6. Unpause Smith's daily cost check after the worker restart, and merge #586 first. Gera's side. Recommend yes.
7. Sync pacing sleep (3c): done as a bug fix, PR #9114; no rate-limit decision needed.
8. CI volume: stop running the full test lane on review events, and cut shards. Needs Gera and Fede together; this is the only lever that gets the runners near $300.

The five questions, answered (written 05:00 UTC; see the audit for what changed since)

1. "Why so b lambda 2000 dollars"

But the instinct was not wrong. Lambda did have its own genuine cost anomaly, and it is the most uncomfortable finding in this document: AWS detected it on Sep 10 — three days before the Config blow-up — and it was never sent to anyone. Lambda ran 548% over its expected cost, $59.81 actual against $9.23 expected, an impact of $50.58. It sat below the alerting bar and was silently dropped. The question asked about Lambda was a question AWS had already answered internally and kept to itself.

What drove it was run time and memory, not traffic: the charge is almost entirely GB-seconds ($54.34 on ARM, $10.91 on x86), with request charges under $1. Functions got slower and stayed busy longer; they were not called more often.

Three functions account for the extra run time. This is duration attribution from CloudWatch, not per-function billing — AWS does not bill per function in Cost Explorer, so treat this as "who got busier", not "who spent what":

The memory and architecture line up with the billed usage types: appfolio-sync (2048 MB, arm64) explains the ARM GB-second charge, stage-replicator (512 MB, x86_64) explains the x86 one. Those two dominate. Why they got slower is a separate open question and is not answered here.

2. "Aws config seems to be the offender" — yes, confirmed

AWS Config is $1,266.69 of $1,935.80 — 65.4% of September. It is the single largest line by a factor of six.

What AWS Config actually is, and why this got expensive

Think of Config as a security camera pointed at the cloud account. Every time anything is created, changed or deleted, it writes a permanent record of what the thing looked like — and AWS charges per record written.

Two settings combined badly:

• The recorder is set to everything (ALL_SUPPORTED_RESOURCE_TYPES) — no filter on what is worth photographing.
• It is set to CONTINUOUS — photograph every single change as it happens, rather than once a day.

On 2026-09-13 at 02:51:14Z the runs-on CloudFormation stack finished creating (verified via describe-stacks, state CREATE_COMPLETE). That stack runs our CI on spot instances: machines are launched to run a build and destroyed minutes later, all day. Each machine is not one record — it is a machine, a network interface, a disk volume and a fleet object, each written on creation, on every change, and on deletion. Config started photographing a machine that never stops moving.

AWS's own anomaly detector put the Config event at 8,200% over expected: $1,070.75 actual against $12.90 expected across Sep 13–15, on the usage type ConfigurationItemRecorded — literally "we wrote a record" — contributing $1,062.03 of the impact. That is anomaly cb999913.

3. The one nobody in the thread mentioned: Security Hub, $188.76

Security Hub bills per security check it runs, and it runs checks in response to Config records. So the record flood produced a check flood, one day behind it. It had been costing $0.00/day through Sep 13, then:

This matters for the fix, not just the accounting. $188.76 of the damage was collateral — a second service reacting to the first. It means turning the Config recording down does not only save Config money, and it means any future Config change has a cost consequence one service over that will not show up under the word "Config". AWS raised its own anomaly for it (65e73754, $29.53 against $10.12 expected, 191.8% over). Nobody was told.

4. "Why did it take so long to detect?"

The honest answer: AWS did detect it. Our side had nothing listening. Every automated guardrail that should have carried this news either did not exist, was pointed at the wrong place, or was set to a bar the event never cleared.

• Sep 10AWS opens the Lambda anomaly (548% over). Below the alerting bar — never sent to anyone.
• Sep 13, 02:51:14ZThe runs-on CI stack finishes creating. CI machines begin churning.
• Sep 13First expensive Config day: $337.69, up from about $3.
• Sep 13–15The anomaly accrues to $1,070.74. A $5/day budget breaches every single day and notifies nobody.
• Sep 14Security Hub joins in at $80.18/day. Its anomaly is also below the bar.
• Sep 16AWS generates the cost-anomaly email — to one personal Gmail address, on a daily digest.
• Sep 17, 04:36ZFede pastes that email into #agent-smith by hand. This is the moment the team finds out. It was a person, not a system.
• Sep 17, 04:51:10ZRecorder fix applied (CloudTrail PutConfigurationRecorder, gera-admin) — 15 minutes after the message. Smith reported it in Slack at ~05:20Z, 29 minutes later.

Four days from the first dollar to the team seeing it — and the only thing that closed the gap was a human reading his own inbox. The response, once anyone knew, took fifteen minutes.

The five reasons, each verified against the AWS API

There are three budgets on the account in total, and all three are breached and all three are tag-filtered. None of them watches the account as a whole:

5. "Change your own code for next time"

That is the literal answer to the instruction, and it is the gap being closed first. The cheap part is that the delivery pipe already exists and already works: the SNS topic propflow-alerts feeds a Lambda called propflow-sns-slack-forwarder-prod which posts into Slack. Anything pointed at that topic reaches Slack today. The runs-on budget was simply pointed at a different, orphaned topic.

Guardrails

(a) Already applied — Sep 17, 04:51:10Z

We have deliberately not closed this ourselves. Live changes to the Config recorder are infrastructure with real blast radius, and the standing rule is that we do not apply them unilaterally. The exact command is in section (c), item 4, and it is waiting on Fede's explicit yes. Until then, treat the EC2Fleet gap as open.

(b) Being built now — a cost watcher inside Agent Smith

A pull request is being opened in agent-smith right now, on branch smith/cost-anomaly-check (agent-smith#583, merged b1ff93a4 on 2026-09-17), that gives Smith the thing it has never had: a daily job that calls ce get-anomalies, and posts anything AWS has flagged straight into #alerts. (Note these are two different rooms: this thread ran in #agent-smith — C0BDW7G1Z62 — while the watcher posts to #alerts, C0BGTNSP04C.)

(c) Recommended, with the exact command — deliberately NOT applied

Each of these is a one-command change to live AWS infrastructure. They are written out in full so a human can read exactly what would happen and run it, rather than approving a description of it.

# 1a. let the AWS billing services publish to the topic
aws sns set-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:194925971150:propflow-alerts \
  --attribute-name Policy \
  --attribute-value '{"Version":"2008-10-17","Id":"__default_policy_ID","Statement":[
    {"Sid":"__default_statement_ID","Effect":"Allow","Principal":{"AWS":"*"},
     "Action":["SNS:GetTopicAttributes","SNS:SetTopicAttributes","SNS:AddPermission",
               "SNS:RemovePermission","SNS:DeleteTopic","SNS:Subscribe",
               "SNS:ListSubscriptionsByTopic","SNS:Publish"],
     "Resource":"arn:aws:sns:us-east-1:194925971150:propflow-alerts",
     "Condition":{"StringEquals":{"AWS:SourceOwner":"194925971150"}}},
    {"Sid":"AllowBudgets","Effect":"Allow",
     "Principal":{"Service":"budgets.amazonaws.com"},"Action":"SNS:Publish",
     "Resource":"arn:aws:sns:us-east-1:194925971150:propflow-alerts"},
    {"Sid":"AllowCostAnomalyDetection","Effect":"Allow",
     "Principal":{"Service":"costalerts.amazonaws.com"},"Action":"SNS:Publish",
     "Resource":"arn:aws:sns:us-east-1:194925971150:propflow-alerts"}]}'

# 1b. move the budget's subscriber off the orphaned topic
aws budgets update-subscriber \
  --account-id 194925971150 \
  --budget-name runs-on-app-daily-budget \
  --notification NotificationType=ACTUAL,ComparisonOperator=GREATER_THAN,Threshold=100 \
  --old-subscriber SubscriptionType=SNS,Address=arn:aws:sns:us-east-1:194925971150:runs-on-AlertTopic-ZGRxpFzLf1Xy \
  --new-subscriber SubscriptionType=SNS,Address=arn:aws:sns:us-east-1:194925971150:propflow-alerts

# 1c. prove it — this must NOT come back empty
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:us-east-1:194925971150:propflow-alerts

aws budgets create-budget --account-id 194925971150 \
  --budget '{"BudgetName":"account-daily-guardrail",
             "BudgetLimit":{"Amount":"60","Unit":"USD"},
             "TimeUnit":"DAILY","BudgetType":"COST"}' \
  --notifications-with-subscribers '[{
     "Notification":{"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN",
                     "Threshold":100,"ThresholdType":"PERCENTAGE"},
     "Subscribers":[{"SubscriptionType":"SNS",
                     "Address":"arn:aws:sns:us-east-1:194925971150:propflow-alerts"}]}]'

aws ce create-anomaly-subscription --anomaly-subscription '{
  "SubscriptionName":"propflow-alerts-immediate",
  "MonitorArnList":["arn:aws:ce::194925971150:anomalymonitor/3aedab90-655d-4a60-9d9f-84a3a9d46e30"],
  "Subscribers":[{"Type":"SNS",
                  "Address":"arn:aws:sns:us-east-1:194925971150:propflow-alerts",
                  "Status":"CONFIRMED"}],
  "Frequency":"IMMEDIATE",
  "ThresholdExpression":{"Dimensions":{
     "Key":"ANOMALY_TOTAL_IMPACT_ABSOLUTE","Values":["25.0"],
     "MatchOptions":["GREATER_THAN_OR_EQUAL"]}}}'

aws ce update-anomaly-subscription \
  --subscription-arn arn:aws:ce::194925971150:anomalysubscription/ebe2a2a4-0c21-452e-991d-66be22bb1117 \
  --threshold-expression '{"Dimensions":{
     "Key":"ANOMALY_TOTAL_IMPACT_ABSOLUTE","Values":["25.0"],
     "MatchOptions":["GREATER_THAN_OR_EQUAL"]}}'

aws configservice put-configuration-recorder --configuration-recorder '{
  "name":"default",
  "roleARN":"arn:aws:iam::194925971150:role/aws-service-role/config.amazonaws.com/AWSServiceRoleForConfig",
  "recordingGroup":{"allSupported":true,"includeGlobalResourceTypes":true,
    "resourceTypes":[],"exclusionByResourceTypes":{"resourceTypes":[]},
    "recordingStrategy":{"useOnly":"ALL_SUPPORTED_RESOURCE_TYPES"}},
  "recordingMode":{"recordingFrequency":"CONTINUOUS","recordingModeOverrides":[{
     "description":"smith-cost-fix-20260916 + EC2Fleet: CI spot-fleet churn drove Config to $1070.75/3d (anomaly cb999913). Daily snapshot suffices for ephemeral CI compute.",
     "resourceTypes":["AWS::EC2::Instance","AWS::EC2::NetworkInterface",
                      "AWS::EC2::Volume","AWS::EC2::EC2Fleet"],
     "recordingFrequency":"DAILY"}]},
  "recordingScope":"PAID"}'

# prove it landed and the recorder is still healthy
aws configservice describe-configuration-recorders
aws configservice describe-configuration-recorder-status

How we will know the fix actually worked

aws ce get-cost-and-usage --time-period Start=2026-09-17,End=2026-09-19 \
  --granularity DAILY --metrics UnblendedCost \
  --filter '{"And":[{"Dimensions":{"Key":"RECORD_TYPE","Values":["Usage"]}},
                    {"Dimensions":{"Key":"SERVICE","Values":["AWS Config"]}}]}'

What success looks like: Sep 17's Config figure back near the $1.53–$5.22/day baseline. What partial success looks like: a large drop that still settles well above baseline — that would be the EC2Fleet gap, and it would make recommendation (c)(4) urgent rather than optional. Security Hub should be checked at the same time, since its charges follow Config's by roughly a day.

One thing not to click

Appendix — all 16 anomalies AWS detected, Sep 1–17

Read from ce get-anomalies on 2026-09-17. The subscription forwards only anomalies at ≥ $100 AND ≥ 40% over expected. Exactly two of these sixteen clear that bar — AWS Config and EC2 Compute — and those are exactly the two services in the email Fede received. The threshold accounts for the delivered set precisely; there is no residual to explain.

The other fourteen all fall below the $100 floor. The largest of them is the Lambda anomaly from Sep 10 that Fede later had to ask about by hand, at $50.58 — barely half the bar, and detected a full week before anyone in the company knew any of this was happening.

Two notes on reading this table. AWS returns no percentage at all for EC2 Compute and EBS because their expected spend was $0.00 — the increase is undefined rather than zero, which is why those cells read n/a. And the Security Hub anomaly only opened on Sep 16, so recency may be a second reason it was not in the Sep 16 email — but at $19.41 it would never have been emailed regardless. Neither note disturbs the finding: the $100 floor let two signals through and silenced fourteen, and the loudest thing it silenced was the Lambda anomaly Fede had to go and ask about himself.

Where the numbers come from