Infra cost review · 2026-08-02 · measurements as of 10:05 AM MT

Cutting the AWS bill ~80% — and the GitHub Actions plan

July's AWS bill was $601.94. 81% of it was DynamoDB, and 97% of the DynamoDB reads traced to two code paths scanning the full production table around the clock. This report gives the attribution, the fixes shipped, what is measured vs. projected, and the GitHub Actions cut plan. Every number's source is stated inline; interim windows are labeled.

July AWS bill — where it went

Source: AWS invoice (account ****1150) + Cost Explorer, 2026-07-01→08-01. CE total matches the invoice exactly.

Item	July $	What it was
DynamoDB reads	367.77	2.94B read units — see attribution below
DynamoDB writes	120.60	Mostly pre-Jul-24 waste (an every-minute sync rewriting ~1,050 unchanged rows), fixed then; write run-rate is now ~$11/mo
ECS Fargate	27.64	Renewal worker (prod 24/7 + a stage service flap-billing while never staying up)
Tax	23.46	Scales with the rest
Lambda	23.44	$15 of it is one function: metric-snapshot's 9× re-fetch fan-out
CloudWatch	11.61	88 alarms + the 5-minute status page's metric queries — not logs (3GB, $1.51)
Secrets/GuardDuty/VPC/KMS/ECR	27.42	No stale secrets found; GuardDuty stays (security)
Total	601.94

The run-rate was climbing, not flat: daily reads quadrupled through July (30M → 196M/day) because full-table-scan cost scales with table size and the table grew. Pre-fix August run-rate projected to ~$729/mo incl. tax.

DynamoDB read attribution — measured

Method: every one-minute CloudWatch datapoint for UTC Jul 30 summed (169.2M RCU, matches the daily rollup); sources isolated by their timing signatures and confirmed against code. One full scan of the 1.17GB table = ~142,700 RCU ≈ $0.53 per scan (validated two independent ways).

Source	Share	Mechanism	Fix
Admin status "spine-orphans" section	49.6%	Two unprojected full-table scans every 5 minutes (288 ticks/day) feeding a drift check whose comparison is day-over-day	Shipped + verified Moved to a daily 09:10 UTC cron; request paths serve the cached snapshot and can never re-trigger a scan (drift-tested)
Stage-replicator anonymization dictionary	~45%	Each Lambda container rebuilds its PII dictionary via full-table scan; containers churn every ~20–30 min, so TTL-based fixes do nothing (tried, measured, ineffective — see honesty notes)	Shipped + verified Sparse GSI live on prod (Aug 2 evening): index created, 11,764 rows backfilled (0 raced, per-cohort counts verified by query), fail-closed row-count floors set on both readers. First post-flip rebuild measured ~750 RCU via the index vs ~142,700 per full scan — a ~99.5% cut on this path, zero fallback events
Everything else (app traffic, syncs, workers)	~3%	Tight queries — including the every-minute reconciler initially suspected, which measured a trivial 24 RCU/run	No action needed

What is measured vs. projected

Claim	Status	Evidence
Spine-orphans scans gone from the 5-min tick	Measured	On-mark minutes fell ~296k → 74–174k RCU immediately after deploy (residual = other status sections). Post-deploy window 09:45–10:00 MT Aug 2.
Replicator scan rate cut by concurrency cap	Measured (interim)	Off-mark mean 83.5k → 35.4k RCU/min in the first post-cap window; short windows stay bursty (a capped container still scans every ~10–20 min until the GSI ships)
Replicator dictionary now reads the sparse index, not the table	Measured	Post-flip window Aug 2 ~21:15 MT: spine-index consumed ~750 RCU for a rebuild that previously cost ~142,700 RCU as a scan; SpineGsiFallback metric zero. Floors: replicator 10,000 (verified 10,510 dictionary rows), app-side per-cohort JSON from backfill counts
TTL-only fix ineffective	Measured (negative result)	9-hour window after raising dictionary TTL 15min→4h: no reduction — container churn defeats it
Post-fix bill ≈ $131/mo; ≈ $89/mo with the small-cuts bundle	Projection	Arithmetic from measured per-source rates. Confirmation needs one clean 24h CloudWatch cycle + 2–3 days of Cost Explorer daily actuals. This section will be updated with actuals.
Two full billed days confirm the cut (≈75%)	Measured (Cost Explorer, settled full days)	Aug 5 check on complete data: Aug 3 billed $4.34 and Aug 4 $5.18 account-wide, vs July's $19.42/day average — a ≈75% reduction, ≈$140–160/mo pace. The earlier "$1.26 / 85–90%" figure was same-day partial data before all charges settled; $4–5/day is the honest steady state. Residual is mostly DynamoDB ($2.70–3.49/day, down from $15.77). Contributor Insights confirmed fully off. Zero regressions.

Small-cuts bundle (applied or queued): stage ECS worker stopped (was flap-billing at desired=1/running=0), prod worker memory rightsize, metric-snapshot 9× fan-out fix, alarm pruning, KMS decrypt caching.

GitHub Actions — $865 billed in July, ~45% cuttable with low-risk changes

Source: GitHub enterprise billing API. Ranking method: per-job durations sampled across 2,274 job records (40 runs × 19 workflows), scaled by exact 7-day run counts — job-level, because matrix jobs bill separately and wall-clock undercounts badly.

Lever	$/mo est.	What it is	Risk
Consolidate sub-minute jobs	82	48% of all jobs run <60s but bill a full minute (one 3-second job bills ~319 min/day)	Medium — required-check names must be preserved
Finish the ARM migration	78	Only CI moved; 83 jobs still on x64 paying a 17–20% premium for one-line changes	Low (hold Playwright + semgrep)
Fold 3 standalone harnesses into CI	68	Full runner setup to run seconds of tests, up to 1,100 runs/week each	Low — same tests, fewer runners
Delete the await-preview sleep loop	53	A job that busy-waits up to 12 min on the meter; its dependent doesn't gate anything	Low — replace with an event trigger
Speed up CI's test job	41	4 separate vitest invocations + a cache key with no fallback (any lockfile change = 100% miss × 19 jobs)	Low–medium

Beyond these: narrowing the renewal-worker deploy gate (~$27/mo) is deliberately not queued — the current always-rebuild behavior is scar tissue from a real stale-deploy incident and needs a design conversation. Same for self-hosted runners (~$173/mo ceiling, real operational costs; a prior attempt was abandoned). Two plausible-sounding ideas were tested and disproven: review-skip runs waste no setup minutes, and superseded PR runs do cancel correctly.

Also fixed: the daily #alerts spend report had been reading the post-discount net and reporting $0 all month while July billed ~$865 gross-of-budget. It now leads with real consumed value, labels the month correctly, and states when the free-minutes pool exhausts (~Aug 4–5 at current pace).

Honesty notes

Three early hypotheses from this investigation were killed by measurement before they reached anyone's roadmap: the every-minute AppFolio reconciler as read culprit (24 RCU/run), the TTL raise as a replicator fix (container churn), and two CI-waste theories (above). The projections table will be replaced with measured actuals after a full clean day; until then the only headline claims made are the ones marked measured.

Fixes shipped as PRs #5273 (spine-orphans daily), #5256 (alerts to Slack), #5283 + #5292 + #5303 (sparse GSI: writers, sync N+1, projection fix — all merged; index live on prod Aug 2). Infra changes applied directly and reversibly: replicator concurrency cap, stage worker stop, Vercel/ECS alert recipient cleanup, reader floors (REPLICATOR_SPINE_MIN_ROWS, SPINE_GSI_MIN_ROWS).