# After merge: deploy, rollback, verification (2026-08-31 to 2026-09-30)
Labels: FACT = fetched; EST = derived. Raw: scratchpad/arch/deploy-post-merge/ and arch/*.tsv.

## 1. Map (FACT, from workflows and runbook ci-and-deploy.md)
| Step | What | Trigger |
|---|---|---|
| 1 | push:main CI: 3 min debounce, superseded runs cancel, affected tests | every merge (5,440 main commits in 30d, ~181/day) |
| 2 | vercel-promote (in ci.yml): create prod deploy via API, wait READY, warm pages, signed-in warm | green main run; waits inside the run for the 30-min window |
| 3 | deploy-lambdas.yml (17 prod fns; 7 alias-flip, 10 not) | workflow_run of CI success |
| 4 | deploy-renewal-worker.yml (ECS Fargate image) | workflow_run of CI success |
| 5 | deploy-announce (Slack), prod-freshness-alarm (cron */30, pages at 2x(build+window)=120 min) | after CI / cron |
| Urgent | [urgent] in title, Revert title, red-main fix, promote-now.yml | lifts window only |
| Rollback | /rollback skill: Vercel instant rollback (web), alias (7 lambdas), old image (worker), revert PR | human, needs Fede go |
| Red main | main-lane-red-alert posts #alerts, freezes auto-merge; red-main-claim.sh picks one driver via Slack claim | automatic + manual |
| Docs | GitHub Action to Cloudflare Pages on push to docs repo | push |

## 2. Verification of prod today (FACT)
- Web: promote waits for READY (catches build errors), warms pages, signed-in warm GET of list pages. No error-rate check, no auto-rollback.
- scripts/canary.sh (curl all pages) is run by hand; only nightly suppression canary runs on a schedule.
- Lambdas: appfolio-sync smoke-invokes the new version BEFORE alias flip. This gate caught 82 failed deploys on Sep 12-13 (bad builds never went live). Real pre-prod canary, only for that lane.
- Worker: "Wait for service stable" step. Nothing checks renewal behavior.
- Freshness alarm: pages only after 120 min behind.

## 3. Numbers (last 30d unless noted)
| Metric | Value | Basis |
|---|---|---|
| Prod web deploys READY/day | median 37.5, mean 40, max 129 (Sep 23) | Vercel API, 561 READY; API only returns READY since Sep 22 (older ones missing), so window = Sep 22-30 only |
| Gap between prod deploys since Sep 24 | median 21 min; 37% under 10 min, 47% over 25 min | 417 READY; batching is not a hard 30-min floor (urgent/revert/red-fix bypass) |
| Commit to web READY | median 18, p75 40, p90 49 min | 561 deploys, commit time from git |
| Vercel build | median 4.4, p90 5.1 min | |
| Earlier audit: merge to prod (full pipeline incl CI) | median 1.13 h, p90 2.46 h | ci-waste-audit-30d.md, EST, 80 PRs |
| Worker commit to deploy done | median 18, p90 28, max 61 min | 101 successful runs Sep 21-30 |
| Worker deploy runs Aug 31-Sep 30 | 1,807 success, 7 failure (Build image x2, Wait for service stable x4, 1 build/deploy) | Actions API counts |
| Lambda deploy runs | 3,446 success, 135 failure: 120 in week Sep 7-13 (82 on Sep 12-13, appfolio-sync smoke before alias flip), 15 after | counts per week |
| Push:main CI | 1,791 success... see weekly: fail 780, cancelled 1,767 vs success 2,828 | weekly.txt; cancel is by design (coalescing) |
| Vercel prod deploy ERROR | 29 (11 in one 10:00-10:40 burst Sep 25, 5 Sep 22, 4 Sep 5, 3 Sep 9/Aug 31) | older ones partial |
| Reverts on main | 17 in 30d; 38 commits tagged [urgent] | git log |

## 4. Prod older than main
- FACT: right now main = 7752f922bf, prod web and worker = 6ca2ebe7c3 (about 1 commit, minutes).
- FACT: Sep 30 the worker sat ~2 h behind web. Cause: each deploy run pins the triggering CI head; GitHub keeps one queued run per concurrency group, so queued deploys for newer SHAs get cancelled. Worker had 161 cancelled runs in the 500 most recent. Fix PR #14289 (deploy main tip). deploy-lambdas still pins workflow_run.head_sha: same bug class, 12 cancelled vs 51 success in last 500.
- EST: web is usually within one window (p90 49 min); worker/lambda lag is unbounded on bursts.

## 5. Incidents CI missed
- Sep 16: Triage voice change, empty post_dial_digits killed every call. Reverted (#8923). Found live, not by CI.
- Sep 7, 10, 13, 15, 22-28: 13 more reverts of behavior/settings; most are design reversals, not crashes (EST: cannot tell from git which were caught by a customer or by Fede).
- Sep 12-13: appfolio-sync bad builds stopped by the pre-flip smoke (a good catch).
- Aug 7 (runbook): failed prod deploy hid behind green commit for 9 h, twice; fixed by waiting for READY.
- No data for how many incidents were found by customers; Sentry not queried here.

## 6. Mistakes (ranked)
1. Deploys pin a SHA, not the tip: worker and lambdas can lag main for hours (concurrency collapse).
2. Nothing verifies prod behavior after deploy; 1 full outage (calls) found by a human.
3. Rollback is human-gated and lane-specific (web instant, 10 lambdas and worker none).
4. Batching does not deliver what it promises: commit-to-prod median 18 min with gaps under 10 min, yet worker and lambda wait on the same chain; latency paid, risk not reduced.
5. Every main commit (~181/day) still runs a push CI whose cancelled minutes are large; deploy latency is mostly CI, not the window.
6. Freshness alarm threshold 120 min is too loose to catch the 2 h worker lag.

## 7. Recommendation (target shape)
Continuous deploy of the main tip, small and verified, instead of batching:
1. One deploy controller: "deploy main tip" per lane, never a pinned SHA (finish #14289, copy to lambdas).
2. Web: Vercel promote with staged traffic (10%, then 100% after 5 min) checked against Sentry error rate and a synthetic call/sms/login probe; auto-rollback via Vercel instant rollback on breach.
3. Lambdas: extend the appfolio-sync pre-flip smoke to all alias lambdas; weighted alias canary (10%) with CloudWatch error alarm and auto-revert. Worker: ECS circuit breaker with rollback.
4. Keep the batch window only as a backstop for cold-start cost (Vercel cold function note, 3.8 s vs 1.7 s); replace with deploy-on-tip plus warm step (already exists).
5. Tighten freshness alarm to 45 min.
| | Batching (today) | Continuous + canary + auto-rollback |
|---|---|---|
| Latency | commit to prod median 18 min web, hours worst case worker/lambda | ~10-15 min, bounded by CI |
| Risk | blast radius 100% at once, human rollback | 10% exposure 5 min, auto revert |
| Cost | low | synthetic probes, alias config, a day or two build (EST) |
| Cold-start pain | fewer deploys | offset by warm step |