CI velocity plan

Written Fri Oct 16 from three read-only inspections of the factory store, GitHub, AWS Cost Explorer and Slack, then fact-checked claim by claim (98 claims, 17 corrected). Nothing was changed. Every number traces to a command in the three reports or to the factory's live scorecard; days are UTC days, the factory's bucket, clock times are MT: merge to live · the Oct 14–15 jump · where the dollars go · the fact-check.

The short version. CI is stable: main was red for only 17 minutes in the last three days (Oct 15, 12:39–12:57 PM), the queue wait is 11 minutes against a 30-minute target, and 94% of PRs merge on the first try. Two things did move. Merge to live tripled on Oct 12 because the production deploy wait was switched from 5 back to 30 minutes. The Oct 14–15 cost jump is mostly more work, not more waste: the swarms merged 1.6 times as many PRs, and the cost per merged PR ($0.69–0.72) stayed inside the previous week's range ($0.51–0.77). The yellow reruns are real but small, and three quarters of them came from two flaky tests.

What moved

Three things the Cost tab does not say

• Nothing is idle. On a normal day, 92% of the machine time AWS bills is real jobs plus their boot. There is no idle fleet to switch off.
• The tab's per-job dollar figures run about 2.5× what the same job costs on a normal day (estimate). Its 7-day window includes the surge, and it deliberately loads the whole EC2 bill onto the two kinds of CI hours it measures, which carry only 51–55% of the dollars. That is how it is built, not a bug.
• A bigger waste than the yellow is counted nowhere: PR checks cancelled mid-run, 385 runs (371 replaced by a newer push) and about $34 over Oct 14–15. Twelve of thirteen sampled replacing pushes were new commits on the same PR, not merges from main.

Top three opportunities

Decisions, one line each

1. Put the deploy wait back to 5 minutes now? Recommend yes. It is a production setting and it reverses the change Gera's agent made on Oct 11 at his request, so it is Fede's and Gera's call, not an agent's. If Gera still wants fewer Vercel builds, the follow-up is a deploy rule that ships the newest tested code when the wait ends: a replay projects about 35 minutes at the same 40 builds a day, but it has never run for real and touches deploy ordering, so it comes after, as its own validated PR.
2. Build the read-only “stop the line” card in the factory app, and allow the one-time 80-run local flaky replay? Recommend yes. No customer or CI effect. The factory app is Gera's repo, so the card goes to Smith on the bridge.
3. Narrow the machine types and remove the unused warm pool, after the placement-score check? Recommend yes, validate first. These are runner-service and AWS config writes, so they wait for Fede's go after the check is posted here.

Still open

• Does a freshly deployed site still load slowly for users? That was the original reason for batching deploys; the reports found no measurement since warm-up steps were added.
• What is the $5–6 a day of data leaving AWS? Likely cache saves and artifact uploads to GitHub (not yet measured).
• Why AWS Config still costs about $2 a day after the September fix.
• The factory's flaky-test list missed the Oct 14 test, and the Reds tab splits one flaky test across 5–6 culprit labels, so this pattern is hard to see on the page itself. The scorecard also showed main 100% green on Oct 15 while the red-spell record shows 17 red minutes.

Sat Oct 17, 2:40 PM MT: Fede chose live rollout over shadow mode. Steps 1, 2b, 3 and 5 ship as small PRs with no switch and a 6-hourly watcher for three days; the fold and the deploy-wait change are dropped; batching (2a) is on hold on Fede's yes at 3:10 PM, because GitHub's queue setting tests every PR separately (no saving, up to 15 min slower; tried Sep 13) and the real saving needs our bot to become the merger, a decision for next week with Gera. Savings on the page are also read as percent of the $142 bill: chip 11–13%, precise tests up to 19%, batching 9–14% (held), checks at arm 4–7%, no second full test 4%. The two paid shadows are off; the selector shadow stays until the queue PR lands.

Live rollout, Sat Oct 17 evening: what is on main now

Percent is of the $150 a day the bill averaged Oct 11–15 (m, factory). Watch so far (Sun 12:55 PM): 4 ticks over 18 hours, 0 red main pushes, 0 missed tests, production level with main, sweeps green every 2 hours, big jobs on the newer chip. One train lost its machine to a spot take-back at 1:58 AM; the retry fired, passed, and the PR merged 10 minutes later. Dollars: AWS cost data arrives about a day late, so Saturday still reads as partial ($138, high because the shadows and bot trains ran until 2:40 PM and we merged ~10 PRs); the first clean day with every step live is Monday Oct 19, readable Tuesday morning. Each line is a small PR with no switch; the watcher checks red-main rate, selector misses, machine families, sweep and promotion, and the daily dollars every 6 hours for three days and a fixer acts on a regression.

Close-out, Tue Oct 20 1:30 PM: three days live, before → after

The watch ended early on Fede's call (it was due at 5:52 PM). Twelve ticks over three days: no red push to main caused by a step, every red merge run either rescued by the spot retry or a real PR conflict, sweeps and releases green on schedule, every job got a machine. One regression (the first batch train stalled) was recovered by hand and fixed forward the same morning. All numbers below are measured (m) from the factory cost endpoint (fetched 12:48 PM Tue) unless marked (e). The Tuesday bill is the first full weekday with everything live; it posts Wed Oct 21.

Verdict. Per unit of work, CI is about 15–20% cheaper and stable. Measured savings at normal volume: about $22/day (m, table below), before batching and the precise selector, which only went live Mon Oct 19 (selector about $2.6/day, m; batching modeled $14/day, e). The plan's $63–68/day is not proven; the Oct 20–21 bills decide it. Open, owned elsewhere: the Fri Oct 23 merge-queue pass-rate check (bar 95%, was 90.2%) is handed to Smith on the engineering bridge; the batch bot's first real red train has not happened yet, so its break-up path is merged but unproven live.

Measured, Mon Oct 19 morning: what the rollout actually saved so far

Everything below is measured (m) unless marked (e). The AWS bill itself cannot show the rollout yet: the last complete billing day is Fri Oct 17, the day before the rollout, and Cost Explorer lags about 24 hours. The first full day with every change live is Tue Oct 20. Rate for every $ line: $0.0253 per vCPU-hour, all-in (Oct 17 AWS CI about $124 for 4,904 vCPU-h, m, next-savings report); at the factory's own EC2 rate ($0.0354) multiply by 1.4. Volume for $/day: the Oct 11–17 average (468 PR runs, 258 trains a day, m, factory). Sunday Oct 18 was quiet (79 PR runs), so a raw $/day compare would flatter us; these are per-unit changes times normal volume. Watch, ticks 7–8 (7 AM and 1 PM Mon): 0 red pushes to main and 0 red trains in the last 6 hours of each tick, sweeps and releases green on schedule, all test jobs on the newer chip. Batching bot live since 9:07 AM after a fix forward (its park step read a field the runner's GitHub CLI lacks); no train formed yet on a quiet Monday. Ticks 9–10 (7 PM Mon, 1 AM Tue): 0 red pushes to main. Every red-looking merge train was a spot machine taken back mid-run; the in-run retry passed and the PR merged on the same train, so none were lost. The spot pool for the checks job was widened to six machine families anyway. The batching bot formed its first real train at 11:57 PM (three PRs). The train caught a genuine conflict between one PR and main, but the bot had no way to break up a train that fails before reaching the queue, so the three PRs sat parked for about an hour. They were put back in the queue by hand and the bot returned to shadow while that gap is fixed.

Sum of what is measurable this morning: about $22/day at normal volume: the two "whole side" rows ($13, which already contain the chip swap, checks at arm and the selector) plus the two push-to-main lanes that sit outside both columns, the re-test sweep (at least $4.9) and the nightly advisory move (about $4, e from the measured per-run cost). The plan modeled $63–68/day. The gap is mostly the lanes that have not had a chance to show yet: batching ($14 e, zero trains so far), checks at arm ($12 e, cancelled by arm-at-open), and the selector. AWS bill, last complete days (m, factory, UTC days, EC2 compute $ / total AWS $): Oct 14 $158 / $202, Oct 15 $155 / $203, Oct 16 $128 / $175, Oct 17 $90 / $138; Oct 11–17 average $111 / $152. Oct 18 and 19 are incomplete and not comparable yet.

Updated Sat Oct 17, 7:05 AM, after the overnight proof fleet (five lanes, five skeptics, one reconciler) and the shadows. Marks: (m) measured, (e) estimate, (r) replay of real records. Evidence, tracking issue #20581. What changed tonight: three dark workflow files and a shadow bot merged to main; the fold shadow was redesigned after a blocking review before it merged. They log only, gate nothing, comment on nothing (the bot creates and deletes its own shadow-batch branches); the required checks on main and the merge guard are untouched; nothing in AWS or GitHub settings changed.

The short version

The bill is a PR counter: 239 merged PRs a day (m), each starting about 117 machines (m) and paying for 2.9 complete test runs (m). Together, after tonight's replays: $142 to about $68 a day, 52% off (e, range $59–80) with 15-minute batches, if agents arm their PRs when the work is done. If they arm at PR open, as today's no-local-type-check rule pushes them to, it is $71, right on the 50% line. The evening model's $63 priced step 3 with the bugs per-push checks catch as free (+$4) and batching from a coarser replay (+$1). The $2,000-a-month line, about $53 of CI/CD, is not reached in any case: it needs about 160 merged PRs a day instead of 239, or a rented server.

The staircase ($/day, all-in, one model, 15-minute batches)

"Alone" is the swing by itself against today's $142; "stacked" is what it still removes after the earlier steps, which shrink the same machines. Only the stacked column adds up: $74. At 30-minute batches the end is $67 (range $58–79).

Each swing: the fence it crosses, what was tried, how we prove it, how we undo it

Before → after by bucket ($/day, model v3, 15-minute batches)

Speed, the secondary goal: where the minutes go and what the staircase does

Measured (speed lane, Oct 7–16): a merged PR takes 61 minutes to go live, an opened PR 116 (medians, m). about 40 of those minutes are the 30-minute deploy window set Oct 11; the days before it sat at 20 and 72.

Decisions, one line each

1. Test once + precise choice: switch the queue to the shadow's list at 0 misses on 10+ real failures, about Tue Oct 20? Recommend yes once it clears; the switch is yours and Gera's.
2. Our own batch bot replaces GitHub's queue once its 30-minute shadow matches the replay's 30-minute figure, then runs at 15-minute windows? Recommend yes, 15 minutes (about $1.7 a day less than 30, 9 minutes less wait per PR); Gera owns the ruleset change.
3. Drop the post-merge re-test for a 2 h sweep, together with the thin deploy trigger? Recommend decide after the test-choice shadow reports; Gera owns the gate.
4. PR checks only when armed? Worth $6.4 stacked only if agents arm when done. Recommend Gera picks Option B or arm-when-done first.
5. Chip canary, 48 h, PR test machines only, automatic revert, Mon–Tue after a weekday 1–2 PM capacity re-read? Recommend yes.
6. Fold the glue once the shadow shows the same verdicts on 500 runs? Recommend yes then.
7. Rent servers? Hold: worth $15–18 after the swings, and the GitHub fee hits it too.
8. Deploy wait back to 5? Recommend yes: $3–7 a day (Gera's bill page) for 42 minutes per change (m). If the Vercel cap must hold, Gera's "ship the newest commit when the window closes" gets 29 of them.
9. Gera's sessions arm auto-merge when the work is done, as yours do? Recommend yes; I post it on the bridge on your yes.

This week

The shadows log through the weekend: the selector's bar lands about Tuesday, the fold's about Oct 26, the bot runs up to 24 dark trains a night. The first night already paid off twice: the bot exposed a defect in the dark train, and the fold shadow found one guard-pin disagreement, explained and its fix merged by 7:20 AM. On a yes: Monday the chip canary and Gera's step-3 pick. At zero misses: switch test selection, then the bot takes the queue, then 2b with 2c.

Risks that cut across everything

GitHub has announced and postponed, with no date, a $0.002-per-minute charge on every Actions minute, self-hosted included: $99–136 a day today (m, market lane), $58–73 after the swings (e, on the evening model). Two hub files are loaded by about 4,000 tests each, so 41% of commits select over 200 test files; splitting them is Gera's. Shadows that run after CI appear as checks on main's newest commit; nothing reads those for a merge or deploy decision at today's window.

Not ours: Gera's open CI PRs, the Vercel build audit (prebuilt deploys possible per Vercel's docs, build-minute billing unconfirmed), ADR-0040, the ruleset, the merge guard.

Written Fri Oct 16 from six read-only lanes: a census of every CI step and machine (Oct 7–15), the graveyard of every attempt since Sep 9 (the factory Log's 112 entries, 48 of them CI attempts from Oct 6–13; 14 session transcripts), Smith's bridge answer, a test compute study, outside references, dedicated-server pricing, and a paired re-read of the Oct 1 one-machine trial. Nothing was changed. Corrected Fri ~3:30 PM: Move 1 re-sized from the trial's paired runs (the first version said $25–35); 3:32 PM: a line-by-line number check applied (72 cells, 11 corrected). Evidence folder. Days are UTC; clock times MT.

The short version. The two busy days were 12% of the two-week bill. The steady state is the problem: about $160 a day of AWS, $0.51–0.77 per merged PR, 239 merges a day. Two moves that change how CI runs, not what it tests, plus housekeeping, take about a quarter off. Coverage trades reach 32–38%. Nothing inside the stability fence reaches half; that takes a different machine model, and the decision is whether to shadow one now.

The bill, and what “half” means

Where the $142 goes ($/day)

The moves

Before → after, by bucket (no dollar counted twice)

Move 2 saves Vercel dollars and 40 minutes, not AWS dollars. Owner trades would take the line to $88–97 (32–38% off). After Moves 1–3 the dashboard would read about $124 a day with product ($102–111 with owner trades), against $160 today and a $67 budget.

What reaches half

Moves 1–3 keep every blocking check and its timing, and land at about a quarter off. Coverage trades reach 32–38% in all and still fall short (the hub trains' full set is the biggest, up to $7–10, and it caught 4–5 real breaks in nine days). Half, and the $67 budget, take the machine model: dedicated or long-lived servers, about $30 a day of rent (unverified price) plus per-job isolation, at the risks in row 5. Gera's Oct 14 line was “only if we need the bill under ~$40/day”; the budget leaves about $53 for AWS CI/CD if the $2,000 is the AWS bill.

Decisions, one line each

1. Deploy wait back to 5 now? Recommend yes. Vercel's $20 monthly build allowance was spent 2.5 days into the Oct 14 cycle, so going back to 5 adds about $3–7 a day of build minutes: a few dollars for 40 minutes per change. Goes to Smith on the bridge; the 30-minute default PR is Gera's.
2. The housekeeping config writes (in-region caches, single runner logging, Config trim, one runner stack)? Recommend yes; each undone in one step, done by the stack's owner.
3. Fold the glue: one PR, job names agreed with Gera first? Recommend yes.
4. The Move 2 spike, one throwaway prebuilt deploy, result to Gera's Vercel audit? Recommend yes.
5. The fork: shadow a dedicated or long-lived fleet for two weeks, now and in parallel? About $900 of server rent plus ~$525 of per-job isolation licence, ≈ $1,425 for the month (estimate; unverified €199 per server, setup fees and minimum term not found); no required checks, nothing in production touched. Recommend yes: nothing inside the fence reaches half, so waiting only delays the one move that does. A purchase, so Fede's call either way.

Not ours: Gera's open CI PRs #20417, #20397, #20365, #20450, #18891; the Vercel build audit; the product DynamoDB read load (~$6 a day of reads, ~$10 of DynamoDB in all).

Rows at target, Wed 1 PM (24 h window): 5 of 6. PRs kicked out of the queue are 20 a day against a target of under 10; the 6-hour readings were 0.

Real AWS compute bill per merged PR: Mon $0.77, Tue $0.35 (Tue still settling).

Real dollars (new, Tue 2 PM). The Factory dashboard (factory.propflowai.co) carries the actual AWS bill. Monday: EC2 $225.92 / 295 PRs = $0.77 of real EC2 per merged PR (Cost Explorer revised it down from the $258 first read): $0.77 against a $0.44 priced estimate (Monday). The estimate counts runner minutes at spot rates only; the bill also carries the runners' disks, boot and idle time, orchestration and other EC2. From this report on, the dollar row shows both, and the $0.70 target is read against the real bill. At the 2 AM (Oct 8) column, five of six rows are at target on the priced estimate; PRs kicked out of the queue (16 a day against under 10) is the one still above the line. Tuesday’s real bill is still settling, so the real dollars row is not called yet.

Real EC2 bill per merged PR (Oct 8). Oct 6: $225.92 / 295 PRs = $0.77 (Cost Explorer revised it down from $258.62). Oct 7: $85.91 so far / 243 PRs = $0.35, still settling.

The window for "now" closed 90 minutes after tonight's fixes merged, so it barely includes them. Merge to live cannot reach 30 minutes while production waits for the second full test run; that is Gera's call (decisions).

A PR now goes from opened to live in about 67 minutes (median), down from 3.6 hours a week ago, and the machine bill per merged PR is roughly a third of what it was. This section is the plain-English week summary; the scoreboard above has the six-hourly ticks.

Six numbers, before and after

Machine time and real bill by day

Source: factory.propflowai.co/cost for hours, AWS Cost Explorer for dollars.

What we did

• Moved CI onto our own AWS machines on Oct 1; 98% of hours run on spot instances.
• Cut the queue's full test pass from 71 minutes to 10 by splitting it into more, smaller shards on bigger ARM machines.
• Stopped the rebuild cascades: a kicked-out train no longer rebuilds everything behind it, and the queue fails fast on a bad entry.
• Made PR checks catch what the queue used to catch: the targeted guard tests now run on the PR itself, and big PRs' tests run in parallel shards.
• Put guardrails on how PRs enter the queue: no re-arming a head the queue already rejected, no arming into a known conflict, a dispatch guard for PRs GitHub never started, and an automatic revert when main goes red.
• A dozen smaller fixes: a security scan that hung for 20 minutes now takes one, a weekly schedule-drift red closed, review-bot noise removed from the queue triggers.

What is left

• Ejections: 16 a day overnight, 0 this morning; the target is under 10 sustained. The remaining ones are merge conflicts between sibling PRs that only the owner can resolve.
• Merge to live below 20 minutes needs the release re-check cut, which is on Gera's side, together with approval of the change that starts test shards 8–10 minutes sooner.
• One hour opened-to-live by Friday.

caution Caution on the dashboard: the cost page's dollar figures for Oct 2–6 are stale first reads and run two to three times above what Cost Explorer shows today (Oct 2: $638 there, $216 in AWS). This summary uses the AWS numbers. The fix is handed to the dashboard's owner.

Close-out, Wed Oct 8. The lane closed Wednesday at about 1 PM as good enough. Open to live went from 3.6 hours to about 72 minutes. Both CI workflow changes merged today: sharded tests start about 8 minutes sooner, and a reclaimed spot machine no longer re-runs a finished train. Two small follow-ups were filed: shard handoff for small PRs, and a daily cost alarm above $0.70.

We checked every number on the Factory dashboard against GitHub and the AWS bill. Counts are good; the timings and dollars Fede would act on are not.

What checks out

• Queue entries and trains building.
• PRs merged in 24 h (250–255).
• Run counts per lane, within 1 of GitHub.
• AWS dollars for finished days match Cost Explorer.
• The live feed is fresh within 2 seconds.
• Slowest-test timings.

Fixes for the dashboard's owner, in order

1. Merged → live counts from the first deploy that contains the PR.
2. Label partial days and UTC days; show clock times with their zone.
3. Price a vCPU-hour from the bill, not a list rate.
4. Blame the first culprit in a failed batch only.
5. Include PR checks in machine hours.
6. Add open → live, the number the one-hour goal is about.

Method: headless render at 1440 and 390 px; every number cross-checked against GitHub and AWS by four independent passes. Evidence is in the lane's scratchpad.

If the Spot retry works, most of the 63 failed-check removals a day should go away. The 35 conflict removals need a different fix: fewer shared files that every PR edits (plan).

The lesson: a red run after merge holds production without telling anyone. Either it pages someone, or production stops waiting for it (decision 3).

The biggest slice is already shrinking (drift tests now run on PRs); the next cuts, in order, are the provider re-runs, superseded PR runs, and the heavy-lane re-test.

Now: make a PR run what the queue will run

Next: stop paying for runs already thrown away

Later: a second queue lane, only when proven

The "now" list was built from Oct 1–3 data, before removals were understood as the main cost. Tonight's three fixes are the new first priority.

Dollar figures are estimates from runner prices (#15799). Depth numbers come from a model that says it runs optimistic, and depth was changed 24 times in a day, so no real day is a clean test.

The real bill (Cost Explorer) lags a day: Monday was $245 of compute.

What this changes: the lane is now judged on minutes and dollars per merged PR, and open to live. The next levers are runs per merged PR, the release re-check, and telling owners when the queue removes their PR (now #17227).

Still open: the depth model's second part is not in a PR yet; the early-failure watcher's idle time is unmeasured; whether disarming survives sessions that re-arm PRs is unconfirmed. Out of scope: runner pool sizing and the deploy ordering wait.