PropFlowAI · CI/CD program · #alerts thread
Deploy model: three options
Proposal — nothing decided, nothing built
For: Gera
From: Fede + Claude session
2026-07-23
The latency program cut PR-open → prod from a 43-minute median to 8.2 minutes measured. The last idle tail is a green, approved PR waiting for a human to press merge. Removing that human (auto-merge, PR #4459, open + on hold) forces a real question, because today merge = deploy: every merge to main promotes to prod within ~4 minutes. Three ways to resolve it below. Gera — this is a proposal for you to poke holes in before we build anything.
Context: where the numbers stand (all 🟢 measured)
| Metric | Before | After | Δ |
|---|
| push:main billed job-min / run | 57–59 | 10 | −83% |
| Type check (deploy critical path) | 240 s | 23 s | −90% |
| Merge → prod promote | 4.8 min | 3.7 min | −23% |
| Worker image build | ~18 min | 3.0 min | −83% |
| Worker deploy wall | 22.9 min | 12.1 min | −47% |
| PR-open → live in prod (canary, end-to-end) | 43 min median (n=144) | 8.2 min | −81% |
One risk that exists today, regardless of option: since #4444, push:main runs the affected subset; the full 12-shard suite only runs nightly. A merge can sit in prod up to ~24 h before the full suite ever validates it. Only option 3 closes that window.
The three options
Option 1
Keep human merge
Status quo. A human presses merge; merge still auto-deploys to prod.
- Zero build work, zero new risk
- A human eyeballs every prod-bound change
- Keeps the idle tail — organic median stays far above the 8-min machine floor
- Every agent/human keeps hand-rolling watch-then-merge loops
- 24 h full-suite blind spot remains
Option 2
Auto-merge + auto-deploy
Merge #4459 as-is: green + approved PRs merge themselves and deploy immediately.
- Fastest possible: ~8 min PR-open → prod, fully hands-free
- One PR to merge; no new infrastructure
- The startup norm — hardenable with post-deploy smoke + auto-rollback + feature flags
- No human anywhere on the path to prod
- Prod deploys on affected-subset evidence only
Option 3
Decouple merge from deploy
Merges go hands-free; prod ships as deliberate, human-triggered releases.
- Merges instant (~8 min); releases gated on full suite + smoke on the exact tip
- Web + worker + lambdas ship coherently; auto-rollback on red smoke
- Closes the nightly blind spot — a safety upgrade over today
- Someone must press ship 1–3×/day or prod drifts behind main
- Red batch = untangling which of N PRs broke it
- Real build work: release workflow, cutover, rollback wiring
What the industry actually does (honestly split)
- Big tech → option 3's shape. Google cuts releases from validated green snapshots of trunk; Meta and Amazon ship staged rollouts with automated rollback. Deliberate release points, decoupled from merge.
- Startups → mostly option 2's shape. The startup norm is continuous deployment: merge = deploy on green, with safety coming from small diffs, feature flags, post-deploy smoke, and instant rollback — not a human release button. It's what most small teams run until scale forces release trains.
- DORA doesn't endorse either button: its elite tier is deploy-on-demand, multiple times a day, and its data favors automation over human gates — small frequent releases correlate with lower change-failure rates whichever way they're triggered.
So the honest fork: option 3 buys a full-suite gate + coherent multi-service releases at the cost of a daily human ritual; a hardened option 2 (auto-deploy plus post-deploy smoke, auto-rollback, and full-suite-on-main instead of nightly) is the more startup-typical path. A hybrid is on the table — that trade is exactly what we want your read on.
If option 3: migration plan — four reversible steps
- (a) New
promote-release.yml (workflow_dispatch): full suite + smoke on tip → promote web + worker + lambdas → release notes to #alerts. Additive; changes nothing until (b).
- (b) Turn off per-push
vercel-promote + worker/lambda auto-deploy triggers. The cutover — done with you and Fede both awake.
- (c) Post-release prod smoke + auto-rollback wiring (web alias re-point; worker/lambda story is open question 3).
- (d) Only then merge #4459 so merges go hands-free. ADR documents the reversal of ADR-0107's deploy-on-green premise.
Open questions for Gera
Pick an option (or a hybrid), then — if it's option 3 — these need your answers:
- Release cadence: pure-manual, or scheduled proposals (e.g. 10am / 2pm / 6pm MT, bot posts "release ready: N PRs — 👍 to ship") with the button still human?
- Who holds the button? Anyone on eng, or a named releaser per day?
- Rollback depth: web alias re-point is instant; what's the right rollback for the renewal worker and lambdas — previous image tag redeploy, or something better?
- Coherent releases: should worker + lambdas always ship with web, or keep their own cadence when a batch doesn't touch them?
- Anything this breaks in your workflows (browser-agent deploys, eval runs, anything keyed to "merge means live")?