CI/CD: where the two hours go, and the big swings

Order chosen by how easily each change is undone, not by size of win. Baseline for every before/after: week of Oct 26, open→live 125 min, merge→live 54, 2% live within an hour, about $75 of CI a day. Reports every 8 hours in #spending, three lines each.

Recommendation: C, in this order. The window and spacing today (one setting and a small PR), then tests at open, then the thin queue with revert-not-freeze, then the express lane and the evidence rule for reds. Dev-tool deletions ride inside the cleanup; the rest of the split waits. Everything in A–C ships without switches, as fixes to live behaviour.

1. Machines are not the bottleneck. A spot machine arrives in 20 seconds; building and type-checking are ~2 min of PR time and ~8 min after merge.
2. Volume is not either. We merge 146 PRs a day now, against 212 a day in mid-September when open→live was 60.
3. The AWS move did not slow us down. Cost per merged PR has been about half a dollar since Oct 5 while the waits were added around it. Four natural experiments say the window decides merge→live:

   Period	Deploy window	Merge→live med / p90	Open→live med	Deploys/day
   Sep 15–24, GitHub machines	none	14 / 20	60	140
   Sep 24–28, GitHub machines	30 min	48 / 72	85	58
   Oct 7–11, AWS + release train	5 min	20 / 28	73	81
   Oct 12–17, AWS	30 min (repo variable)	61 / 80	116	41
   Oct 17–29, plus tests-on-arm and the 2-h sweep	30 min	53 / 77	120	36

4. The merge queue re-test catches little, not nothing corrected Oct 30. The first count (54 failures, all spot reclaims, all merged unchanged) looked only at runs the queue let through; a rejected run ends as cancelled. Recount by the Opus judge: about 50 rejections in 36 hours across 40 PRs, 11 of which needed a fix before merging, and at least 5 were breaks that appear only against current main (3 caught by guard pins, 2 by type-check or build). So the thin queue keeps guard pins plus type-check, lint and build, and drops only the test re-run. The 7-day recount (Oct 30): 65 real rejections, 35 by guard pins, 27 by type-check or build, 3 unit-only (a flake, a spot reclaim, a closed PR), so a queue that keeps guard pins would have missed none. Its saving is about $6–9 a day, not $20–26.
5. Rework multiplies everything. The median PR gets 3 CI runs; 35% of runs are cancelled by a newer push; 35% of PRs have a red run. Each push after open restarts the ~21-min armed run: open→merge is 41 min with no pushes, 57 with one, 78 with two. Even a clean PR (38% of PRs) takes 101 min open→live, so the fixed waits have to shrink.
6. Gera and Fede see the same pipeline. Last two weeks: same size, same review rounds, same red rate, same queue and deploy times; 125 vs 120 min open→live. Gera's PRs wait 27 min to arm; Fede's arm at open and sit 45 min in the armed phase, about 18 of them parked by batch trains (live since Oct 20, though the census called them dark; 0 of 33 of Fede's PRs since Oct 28 rode a batch). Draining the trains fixes Fede's side; tests at open fixes Gera's.

On #1, the Vercel doc from Oct 11 prices the window at $5.55–$9.72 a day; that puts Vercel at $15–19/day against a $6.67 budget. If we want Vercel flat later, build the production bundle once on our machines and only upload it (also removes the 6-min build), at the price of production secrets on CI machines: a security call. Faster machines are on no list: the whole checks job is under 4 minutes.

Today there is no way past the merge queue: the ruleset has no bypass list, editing it empties the queue, and GitHub's "jump to the front" option exists but is unused. [urgent] only skips the spacing and the window after merge (measured 8 cases: 41.5 min to merge, 20 to live). The manual promote failed 3 of its last 6 runs on a short commit id.

Guard, so it never becomes the normal path: an express label plus an [urgent] title and an incident link (who applied the label is not a guard: every session acts as Fede); no change under auth, middleware, permissions, secrets or the merge guard; at most 3 a day, each posting one line to #alerts; the full sweep runs on the shipped commit and a red one reverts it automatically; a weekly count on the Decisions page; fix the manual promote to accept a short id or a PR number.

The repo was inventoried unit by unit: files, lines, whether the unit is in the web build, the type check and the test set, and which workflow deploys it. Compile and type-check are about 2 minutes of a PR run and about 8 minutes after merge (queue build 1.8 min, Vercel build 5.5 min). Those are the only minutes a split can touch. None of the splits touches the review, queue or deploy waits that make up the two hours.

Fede asked for this on Oct 29; it was parked as Later behind option C. Tonight two one-line homepage copy PRs each ran the full app CI, so it is a go.

Also found: mock-pms-service, testing-harness and tools are type-checked for no reason; two stale pages ((public)/test, api/mock) can go; 8 packages have no importer. Seconds each, zero risk.

Moving the marketing app (marketing.propflowai.co) out Live — Oct 31

Decision (Fede, Oct 31 00:1xZ): option A, with login, no MFA, "like how we did factory." Cut over the same night: the dashboard now lives in its own repo and Vercel project (propflow-marketing), and marketing.propflowai.co sits behind a Cloudflare Access app with Google sign-in for any @propflowai.co address (same shape as factory.propflowai.co; the app itself refuses any request without a valid Access token). The visits number reads the main app's Vercel Analytics project, so it matches what the dashboard showed before.

Still open: one real Google sign-in on the new host by a person confirms the six views (every automated step is proven: edge redirect, origin reachable, token audience matched, six views on the preview build); if it fails, rollback is Revert #24082 + DNS unproxied + domain back to the app project. Also open: whether the dashboard keeps the copied read/write AWS key or gets its own read-only user (Fede's call); rollback = set the DNS record back to unproxied and move the domain back to the app project (two API calls, under a minute). Scoping that preceded the decision is kept below.

The scoping (Oct 31, 00:00Z)

What it is. One staff dashboard at marketing.propflowai.co (Overview, Traffic, Channels, Campaigns, Leads, Connections), about 1,000 lines of page code plus 1,500 of data and LinkedIn code, built Oct 24, open only to the three platform admins after sign-in and MFA. It reads visit events and campaign counts from the traffic table, demo requests and the LinkedIn token from the main table, and the Vercel Analytics and LinkedIn APIs. No cron feeds it.

What cannot move. The visit-recording endpoint the public pages post to, the /go campaign links and their counters, the code that writes traffic rows, and sign-in. Only the dashboard (the read side) can move; the one shared contract is the traffic key layout.

Change volume: 13 commits in 60 days, 8 of them on the day it was built; nothing since. A move buys ownership isolation, not publish speed. Recommendation: C; B if Fede wants it out for ownership; A only if a non-engineer must ship dashboard changes alone. Unverified: whether the app's session check accepts a forwarded cookie from another server; the traffic table's access role.

Marketing site as its own repo: the plan Proposed — pending review

Tonight's two copy changes each paid the whole app pipeline: type check, lint, build, test shards, paid bot review, merge queue. One needed a 19-minute run after two cancelled ones; the other went red for 13 minutes and re-ran. The case is speed to publish and keeping copy edits away from the app, not CI spend: only 3 of 998 PRs merged Oct 16–30 were marketing-only (~42 machine-minutes in 14 days).

What makes A work. 37 site files import the app's UI kit, 28 import app library code, 2 touch login-like paths. Those are what get copied or cut.

Rollout, dark and small.

1. List every route served under propflowai.co today; mark each site or app.
2. Create the repo and Vercel project on a preview address; copy the public pages and the UI pieces they need; make it build.
3. Screenshot every public page against production at 1440 px, headless, and diff.
4. Cut propflowai.co over at a quiet hour, with an instant swap back. Fede's go; turning the root domain on is his call.
5. Delete the public pages from the app repo in one PR.

Risks to carry through: identical URLs for search ranking, analytics tags (they live in the app shell today, not in the site code), the [preview] build flow for unpublished designs, and who owns the site repo (Fede's sessions or Gera's).

Footprint: src/app/(public) 100 files / 24,874 lines; src/components/marketing/page 77 / 21,314; marketing/components 12 / 10,741; marketing/home 29 / 1,865; 21 tests / 2,222 lines; 37 assets under public/marketing and public/videos. (standalone)/marketing is an internal analytics dashboard and stays. No site repo exists today; camellia-site is the nearest shape. Tonight's copy PRs: #23897, #23899.

Gera's side (Agent Smith, Oct 30 22:15Z): agrees the end state is a separate repo with its own Vercel project, but recommends two steps. Now: a CI filter so a PR that only touches marketing paths runs lint, a build and the bot review, then goes live without the full test suite. Then: the separate repo. His coupling read matches ours: the marketing tree pulls brand tokens, the home content module, shared UI and composition pieces and the logo animation; about 12 tests and the middleware (which decides what the root domain serves) depend on it, so the root route and the preview links move with the site. He found no earlier written decision either. He asked that the two auth-like imports be named on this page.

Stopped, Oct 31 00:35Z: Fede: Gera is already driving the landing page refactor, so this lane ends here. What exists is handed to Gera through Smith: the repo, the idle Vercel project, the merged dark app change (inert while the setting is off), and the cutover issue. Nothing customer-facing changed.

Built, Oct 30 23:10Z. Steps 1–3 done: route list (21 site, 24 app, 13 shared), repo propflow-site with its own Vercel project on a preview host, 27 pages pixel-identical to production (one animated icon differs by 0.002%). Shape changed to multi-zone: the app keeps propflowai.co and every app path; only the 27 marketing paths get sent to the site, and only when a setting is on (off = today's behavior). Found and fixed before any traffic: cache header parity, "don't index" on the preview host, assets served from the site's own host, and the security header that lists allowed asset hosts (both repos). The app-side change is PR #23963, merged dark Oct 31 00:0xZ: nothing changes until the setting is on (cutover issue #23957). A throwaway probe proved a Vercel rewrite stamps the forwarding domain on the request, so the live pages will stay indexable. Cutover ask is with Fede. One slip: the traffic beacon was live on the preview host during the first screenshot runs; being counted, nothing deleted without Fede.

Decided: A — Fede, Oct 30 22:30Z ("go, you have 4 hours"). Build window to about 02:30Z Oct 31: new repo, own Vercel project on a preview host, screenshot match of every public page against production. Constraints: every domain accounted for, nothing breaks, analytics identical. The root-domain cutover is a separate go from Fede.

Smith's two-step order (B now, then A) is on the table as a fourth choice; the trade is a faster copy lane within days against building a path filter that is thrown away when the repo moves.

• Dead, delete: the old auto-merge logic (3,522 lines, off since Oct 1), the review re-check and rule-9 re-check (off), four shadow measurement lanes, branch preview domains, two runner canaries, two never-run A/B lanes. 13 files, 6,351 lines.
• Red and nobody reads them: the database-escape sweep (6% green), the weekly stress run (0%), the nightly renewal canary (52%), the renewal scoping evals (33%). Their only output is an 8-hour report whose schedule is commented out. Fix with an owner, or delete.
• Spend with no consumer: day0-replay runs ~650 times a day writing an artifact; the pre-queue type check runs ~379 times a day and nothing required reads it; the behavior-eval pack was dispatched 75 times in one day by one actor, 84% red.
• Machines started only to wait: 8 crons at 10–30 min intervals start ~600 machines a day for sub-minute jobs; the deploy job holds a machine ~30 min per deploy while Vercel builds (~1,450 machine-min/day).
• Review plumbing: each review verdict starts four more workflows (~83,000 runs in 30 days); six workflows can write the required review check.

Where the machine minutes go on a day like Oct 28 (1,495 CI runs): PR checks and tests about half, the merge queue's per-PR re-test about a third, main-branch runs and the 2-hourly sweep the rest; scheduled jobs add ~1,640 machine-minutes. The queue cut (#2) saves $20–26 a day, tests-once (#3) $9–18, the cleanup (#4) $5–7: together $35–50 a day, from ~$75 to under the $65 target. (Dollar deltas are shares of the AWS bill, ±30%.) Per-run price is already 6× below GitHub's; what is left is runs per PR, not price per minute.

• Live = the first production deploy whose commit is at or after the PR's merge commit on main (Vercel records the GitHub deployment at success; 32 of 32 sampled within 3 s). Same definition as the factory app.
• Weeks are by merge date, Monday start. GitHub-hosted runners until Sep 28; our machines Sep 29; merge queue Oct 1; release train Oct 4; deploy window back to 30 min on Oct 11, 9:25 PM CT (repo variable PROMOTE_WINDOW_MINUTES; the script default and the runbook still say 5); tests deferred to arming Oct 17.
• The 119-minute breakdown uses the timelines of 3,259 PRs (arm, queue, merge events) joined to deploy runs; minute budgets inside jobs come from ~60 CI runs, 25 queue runs, 15 sweeps and 15 deploys (Oct 19–30). Medians with those sample sizes.
• Dollars per move are the AWS bill split by machine-time share (±30%), not metered readings. The review bot's token cost is not in them. The "CI on AWS" line is EC2, EC2-other, CloudWatch and VPC above the Sep 22–28 baseline.
• Review rounds and pushes move together in the data, so they are confounded. Red-sweep freezes affect the worst 1% of PRs, not the median.
• Pulls: every merged PR (per day, to avoid the 1,000-row cap), 48,478 CI runs, 3,712 sweep runs, 3,113 production deployments, daily AWS Cost Explorer. Raw files and scripts stay in the session scratchpad; a memory note records the recipe.
• The factory app (factory.propflowai.co) sits behind a login wall and its database refused the session's tokens; nothing here was read from it.

CI runs per PR is 3 since Oct 17 because arming now starts a second run by design.