THE TARGET (locked by Fede, Aug 10): 15 minutes from a change being opened to serving in production. STATUS, Aug 11: hit at the floor. Both end-to-end probe changes served in production in 14 m 54 s and 14 m 52 s — measured on the real pipeline with everything enabled, including the probe that a sibling's merge would have cost 65 minutes the night before. The median real change still takes hours, and that residue is now almost entirely the review conversation — the review-layer fixes landing tonight are the next bite. Hard constraint unchanged: the planted-bug catch rate of the review layer holds or improves — and it did: the benchmark was rebuilt (8 stale cases → 12 verified-real, scoreboard now committed) and the shipped prompt change measured 2/4 → 3/4 on it with zero regressions. Progress reads out automatically in the daily 6 p.m. email.
| Aug 10 | Aug 11 | |
|---|---|---|
| One change, opened → merged | 7 m 34 s | 6 m 55 s |
| Same change, knocked stale by a sibling's merge | 24 m 45 s | 5 m 21 s |
| Opened → serving in production | 29 m · 65 m | 14 m 54 s · 14 m 52 s |
| Production build | 13–18 min | ~3 min |
| Test lap | 8.3 min | 4.3 min |
| Green builds that never deployed | 40% | 0 observed |
| Catch-up laps · stuck checks (twin probe) | 1 · 1 | 0 · 0 |
| CI laps burned per change (probe) | 5 | 3 |
| Test runs killed by review comments | 64% cancelled | skipped at zero cost |
| Median real change, opened → live | unmeasured | 5 h 40 m · daily 6 p.m. email |
All real production runs, Aug 10 vs Aug 11 · quality guard held: planted-bug recall 2/4 → 3/4, zero regressions · full evidence below.
Shipped and verified live: the re-check rule is off (pick 1) · test lap split six ways · deploy hand-off (never silently skip) · "production is behind" alarm (watched two quiet cycles, no false page) · build cache working again · bigger build machine · reviewer prompt served from the trusted branch · daily opened→live number in the 6 p.m. email · MFA clock-change bug fixed · dead nightly type-checker safety net revived.
Review-layer fixes — shipped Aug 11: nits and advisories never block a merge again, at any round (validated first: planted-bug recall improved 2/4 → 3/4 with zero regressions and no finding-volume inflation) · phantom review events now boot zero test runners and can no longer cancel the run carrying the verdict · the planted-bug benchmark rebuilt from 8 stale cases to 12 verified-real ones with a committed scoreboard. One candidate was rejected by the evidence: batching findings into one review event is impossible in the reviewer's sandbox — the reviewer itself proved this in a blocking verdict that caught the change silently deleting every inline finding; that PR is closed with the record, and its one salvageable piece shipped separately and verified live: duplicate verdict posts on yellow rounds are gone — 1.32 verdicts per round before the fix, 1.00 across every round since, zero doubles observed.
Next bites, scoped but not started: top target: the reviewer ran clean but posted nothing three times today, each costing a manual re-fire — and the failed check a silent run leaves behind blocks merging even after a later genuine approval (a third misleading-check source, distinct from the two fixed/explained today) · the reviewer's round-memory repair (re-fires and rebases still wipe its memory and buy a full fresh review) · the date-arithmetic rule's missing duration-vs-calendar exception · the fix-up-chain completeness clause.
Known gaps, on the list: a change can still merge on a stale green while a newer test run on the same change is mid-flight — observed once tonight, and it put a red onto the main line for ~40 minutes · the "hold" label did not stop the repo's own auto-merge sweeper from merging five labeled changes — the label is not the barrier it appears to be · with the re-check rule off, changes no longer re-test against the exact code they land on; that is the priced trade from the lab experiment, now backstopped by the post-merge full sweep, the freshness alarm, and probes · very large changes (over the 200-test-file cap) don't re-run their full suite after a new push — they wait for a fresh review verdict, which reads as a "stuck check" (tonight's ghost-check scare was exactly this, misdiagnosed; no regression existed) · two different workflows can write the same required check name (a docs stub and the real test lane) and which one wins is unpinned · the freshness alarm can be falsely reassured (never falsely alarmed) while a deploy is mid-build · 30 high-severity dependency alerts predating tonight · 164 old work folders holding unmerged work need a judgment sweep.
Completed results only — estimates and in-flight runs don't appear here.
| Result | Number | How we know | Date |
|---|---|---|---|
| Cost of the re-check rule on an identical change | 3.3× slower (7½ min → 24¾ min) | Two identical tiny changes sent through the real pipeline; only difference was merge order | Aug 10 |
| Speed when nothing is stuck | ~40 s approved→merged | Timed on real merges | Aug 10 |
| How much slower August got vs July | 26 → 139 min typical · stuck-over-3h: 16% → 46% | Every merged change, both months | Aug 10 |
| Reviewer asks per change, July → August | ~3 → ~13 | Counted on 50 changes per month | Aug 10 |
| Deploy-failure root cause found: the build runs out of memory on the current build machine — randomly, because it sits at the ceiling | 11.3% of deploys failed (4 days) · same code failed 2×, passed 3rd try | Every failed deploy Aug 7–10 matched to the deploy platform's own error records | Aug 10 |
| Production served outdated code since Aug 7 | ~21 h total (11.6 h of it silent, before alerting existed) | Failure→next-successful-deploy windows, verified per commit | Aug 10 |
| Do the background-worker deploys add waiting? No. They trail the website by ~9 min (nobody waits on them); the cost is a ~9-min consistency gap per merge, and no undo path exists for them (undoing a bad worker deploy ≈ 17–22 min of roll-forward) | median +4.5 m (jobs) / +8.7 m (worker) · 100% deploy coverage since Aug 3 | 448 green merges' deploy runs, Jul 30–Aug 10 | Aug 10 |
| Does merge→live contribute to waiting? Yes — it owns the catastrophic tails. Typical day it's the minority leg, but it has quietly gotten ~5× slower over five weeks, and the main cause is not the crashes: 40% of green builds never deployed at all — each deploy run stepped aside for a newer merge that also stepped aside, so under merge pressure nobody ships and no alarm exists for that state | 17 min median (30 d) → 29 min (last wk) · worst merge waited 19.1 h · prod behind main 49% of all wall-clock over 30 d | 1,888 merges matched to actual serve-time via deployment records (cross-validated to 0.5 s) | Aug 10 |
| Where the 8-minute test lap actually goes: 70% is a fixed safety-net suite (366 test files) that runs identically on every change — the part that depends on the change itself takes 1 second. The required “Build” check builds nothing on PRs (a 4-second placeholder; the deploy platform does the real build). Queue wait is ~3 seconds flat all day — bigger or more machines would buy nothing. And 30% of all CI spend is runs triggered by review comments, 64% of which get cancelled mid-flight. Sharding the safety-net suite the way the post-merge suite already shards is the one move within an order of magnitude of the others: it attacks 5.9 of the 8.3 minutes | 8.3 min median lap (9.8 p90) · 5.9 min = fixed guards · ~31% of ~9,100 daily runner-minutes is cancelled work · 16 CI runs consumed per merged change | 80 successful laps decomposed job-by-job, step-by-step, from 5,500 real runs over the last 7 days | Aug 10 |
| The safety-net suite can shard with zero coverage change — measured, not modelled: the 366-file block runs 126 s on one worker locally, 26 s split four ways, 9–12 s split twelve ways (the post-merge suite already runs 12-way in production daily, so the mechanism is proven). Bonus finds: the change's own tests take 3.8 s; the test runner boots three times per lap (~32 s each) where once would do; and file loading costs twice as much as the tests themselves | test lap 8.3 → ~2–2.5 min with sharding · guard set grew 55 files in 10 days | Full suite profiled file-by-file on a local copy; shard timings measured directly; 60 evidence files saved | Aug 10 |
| The deploy build is the real moonshot blocker: 15.8 minutes, median, waiting for the platform to build — the same build runs in 73 s cold / 26 s warm on a laptop. Root causes measured: four code sites that read files dynamically force the bundler to trace 15,126 files into many bundles (same mechanism as July's function-size incident), the build machine is throttled to 2 cores because more cores hit its memory ceiling, and the build cache was removed in May when it held <1 MB — it holds 1.2 GB today and is worth 47 of those 73 laptop-seconds. That removal was right then and is wrong now | floor opened→live: ~33 min today → ~26 with tests sharded → ~17 with the build also fixed | 14 real production builds timed via the deploy platform's own READY signal; local builds timed cold and warm | Aug 10 |
| Quality hole found while profiling: the nightly safety net that proves our fast type-checker matches the reference one has been crashing for weeks (out of memory — needs 7.4 GB, capped at 6) — so nothing currently proves the fast checker catches what the reference catches, and every fast lane relies on it. Also found: one test that fails by exactly one hour twice a year (daylight-saving bug). Fixes queued | 4 consecutive nightly failures · one-line memory fix | Nightly run logs read directly | Aug 10 |
| The test-lap shard is built and measured on real CI: 8.3 → 4.3 minutes, reproduced twice, zero coverage change — all 366 safety-net files still run, split six ways, with tamper-proofs (a shard that collects zero tests fails; a manifest counts and fingerprints the full set before any shard runs). Held for one-click merge, alongside two more held fixes: the triple test-runner boot collapsed (~32 s per lap) and the dead nightly type-checker safety net revived (its "16 GB runner" premise was false — the machine has 7.9 GB, and the job now prints real capacity so the next decision reads measurements). Bonus with teeth: the daylight-saving test failure was a real product bug — MFA lock expiry ran off by exactly one hour whenever its 90-day window crossed a clock change, invisible in CI which runs on UTC. Fixed and already merged to production the normal way | lap 8.3 → 4.3 min measured · 3 held PRs ready · 1 product bug fixed+shipped · a 49-second sleeping test now runs in 0.7 s | Measured on the shard PR's own CI runs; every PR bot-reviewed (3 rounds each, real catches every round) | Aug 10 |
| The deploy build collapsed in production tonight. Four code sites that read files dynamically were forcing the platform to bundle ~6,800 spurious files into every one of 531 serverless functions. Fixed, reviewed, and merged the normal way — and the first production deploy after it spent 1 m 56 s on the phase that had been running ~10 m 22 s (one deploy so far; not yet claimed as the new median). A second, held fix ends a quieter absurdity: the build cache is created and thrown away on every single deploy — it exceeds the platform's size cap, so the expensive half never survives (confirmed on 6 of 6 consecutive deploys) | deploy-output phase 10 m 22 s → 1 m 56 s · whole deploy 15.8 min median → 6 min on the first post-fix deploy · bundled project files −91% | Before/after read from the deploy platform's own logs; 14 prior deploys as the baseline | Aug 10 |
| The before/after replay is done — real changes, real pipeline, one rule flipped. Two of yesterday's actual merged changes were replayed through the lab's full pipeline twice: once under today's rules, once with the re-check-staleness rule off. Review quality was identical in both arms (one reviewer round each). Under today's rules, both went fully green and were then blocked purely for being stale — each paid one to two complete extra laps, and one got knocked stale a second time by its sibling merging (the compounding treadmill, reproduced live). With the rule off, the identical changes merged on their first lap. The traded risk, stated plainly: they merged while 11 commits behind the latest code — that exposure is what the rule buys, and this experiment prices the rule, it doesn't decide it | open→merged 24.0 → 5.4 min and 14.9 → 5.4 min · CI cost per change −70% / −58% (64→19, 43→18 machine-minutes) · sync laps 2→0 and 1→0 · review rounds 1→1 | Full A/B in the lab (same CI, same AI reviewer, same merge rules as production), per-change lifecycles recorded; lab verified restored to baseline after | Aug 10 |
| The lead-time number now ships in the daily 6 p.m. email, computed by the machine (pick 5 — adopted). One locked definition: raw open→merged, median, with share-over-3h as its own headline; the two competing filtered definitions were deleted, and the old ≤3h filter would have hidden the entire August regression. First run also shows merging itself takes 30 seconds once green — the whole wait is getting to green, across ~17½ reviewer rounds per change | first daily block: 5 h 30 m median · 63% of changes over 3 h (24 h) · 30-day median is 31 min vs 4 h 46 m last week — the slowdown is recent and steep | New metrics script merged through full review (26 tests pin the definition), run against live data | Aug 10 |
Nothing is in flight — every result above comes from finished, verified work.
The field research mostly confirmed we are ahead of 2026 practice (review on subscription, previews off by default, standards-as-code, two-tier CI). The real losses are internal and measured: August's raw median open→merged jumped from 26 min (July) to 139 min, and the share of PRs stuck >3 hours went 16% → 46% — while the machinery itself still moves a green PR in ~40 seconds. The lead time is not in the gates; it is in re-verifying unchanged code, checks that never conclude, green PRs nobody re-arms, and review loops that don't converge. Four of the five moves below need no new vendor and no new spend — one is literally finishing a decision we already made.
| Fact | Number | Source |
|---|---|---|
| Merged PRs, last 30 days (≈63/day; peak 138 on Jul 30) | 1,888 | GitHub Search API, 2026-08-10 |
| Median open→merged, July → Aug 1–10 (raw) | 26.3 → 138.8 min | gh API, all merged PRs/window |
| PRs taking >3h, July → August | 16% → 46% | same |
| Bot review rounds per merged PR, July → August (mean, n=50/window, directional) | 3.0 → 13.1 | gh reviews per PR |
| Open-hours spent after the final LGTM (idle: holds, unarmed green PRs) | 51% | pipeline-delivery.html §3 (21d, 1,289 PRs) |
| Share of open-hours held by the worst 8% of PRs | 67% | same |
| Branch commits that are merge-from-main syncs (the BEHIND treadmill) | 39% | 148/378 commits, wk of Aug 3–10 |
| Required “Unit Tests” check-runs stranded with no conclusion (median 94 min to clear) | 37% | 56/152 check-runs, issue #5534 |
| Green review checks carried forward from a different commit | 54% | 96/179 check-runs (D2) |
| PRs needing a manual “@claude review” nudge (~500 nudges in 3 weeks) | 39% | pipeline-delivery.html §3 |
| Auto-merge disarms vs arms (the machinery un-arming itself) | 105 : 39 | 30 PRs, same doc |
| Machinery floor when nothing blocks: green→merged / open→first-verdict | ~40s / 1.0 min | same |
Reading: the middle of the distribution is fine. The lead time lives in waiting states — exactly the factory read: WIP piling up between stations, not slow stations.
[preview] builds stay ~zero (Gera). One correction from the session-history dig: full-suite coalescing was listed here as “rejected by Fede” — the written record says rejected, but the session logs show the question may never have actually been answered (it was one of two decisions put to Fede that day; only the other got an answer). It moves to the decisions below rather than staying silently “settled.”What: Classic branch protection on main still has strict: true (“require branches up to date”), which forces every open PR to re-sync and fully re-verify after every merge (~30/day). ADR-0107 already decided against this — verbatim: “strict: false (requiring up-to-date branches would reintroduce the re-run treadmill; the post-merge run covers stale-base drift).” The cutover flipped the ruleset (which is strict: false today) but never touched the classic layer. The #1 measured bottleneck is a decision we already made, applied to one of two gate layers.
Cost of the miss (measured): 148 of 378 branch commits in the measured week — 39% — existed only to re-sync; each re-run costs a mean 17.9-minute CI lap. Each sync also disarms auto-merge (105 disarms vs 39 arms), invalidates the review verdict (feeding the 54% carried-verdict rate), and re-strands the Unit Tests umbrella — the living doc calls this “one loop, not four independent bugs.” Worst case on record: #5631, 23 review laps + 11 syncs.
Measured 2026-08-10 (controlled probe, two identical one-line PRs opened 8 s apart): the one that merged first took 7 m 34 s open→merged; the identical twin, knocked stale by that merge, took 24 m 45 s — a 3.3× tax (+17 m) from re-syncing, re-running checks, losing its armed auto-merge, and needing a manual nudge to restart its test lane. Armed + approved + green + stale does not self-heal: the machinery never updates a stale branch on its own. Source: probe harness PR #5645, results JSON in the run record.
Lab-proven 2026-08-10 (full A/B, real replayed PRs): with the re-check rule off, the same two real changes went 24.0 → 5.4 min and 14.9 → 5.4 min open→merged, at −70% / −58% CI cost, with review rounds unchanged (1 → 1 in both arms). The traded risk: they merged 11 commits behind. Full lifecycles in the lab results record — see the outcomes table.
Do: flip classic protection to strict: false (one API call), or better, collapse the two overlapping protection layers into one ruleset so there is a single definition of the gate. Then re-measure the week's sync-commit share and disarm ratio. One flag to verify first: the current-state map claimed strict closed the #3521 clobber class, but the archaeology (which read the actual PRs) found #3521 was a stale-worktree snapshot bug fixed by docs-only-fence.yml, not by strict — and ADR-0107 itself says the post-merge run covers stale-base drift. Verify the fence covers the class, in a sandbox repo if needed, before flipping.
Imagine a bakery where, every time any cake leaves the oven, every other cake in the room must be re-weighed, re-tasted, and re-approved from scratch — even though nothing about them changed. We wrote down months ago that this rule should be off. It's still on in one of the two rulebooks, and it eats about 44 hours of waiting a week. Turning it off is minutes of work; the care is making sure the one real accident it was believed to prevent is actually prevented by the separate fix we built for it.
What: Three known defects make required checks silently never conclude: the hand-posted Unit Tests umbrella can park in_progress forever (D1 — 37% of check-runs, median 94 min, 14 never resolved; issue #5534 calls it the highest-yield fix on the board); a cancelled run deliberately writes nothing, preserving the stale pending state (D8 — ~47% of ci.yml runs are cancelled); and on workflow-only PRs the review check attaches to the wrong ref entirely, so the repo periodically loses the ability to repair itself (D4 — the Aug 9 fix required disabling branch protection). Every stranded check today is resolved by a human noticing — that's the 500-nudges-in-3-weeks tax and much of the 2.7:1 disarm churn.
Do: one concern, three fixes: (a) a terminal-state guarantee — the reporter always posts a conclusion, and a cancelled/superseded run posts something (“superseded by run N”) instead of nothing; (b) apply #5622's ref-attachment fix (already done for Unit Tests) to the review check; (c) fold in the D12 flaky test (outlook-webhook, fails ~1 in 3 standalone) that randomly reddens unrelated PRs. Field-check: quarantine tooling for exactly this is free at our size (Trunk — bot commits don't count as committers), though a targeted fix may be simpler.
Measured 2026-08-10 (probe): the stranding was reproduced live, but the mechanism is sharper than first written: on small diffs it is not “a push interrupts the test lane mid-run” (too fast to interrupt — that narrow claim is refuted at small sizes). The two real mechanisms, both observed: (a) after a stale-branch sync, the required test check goes absent entirely — no review event fires, so the lane never starts; the probe sat wedged 7 m 08 s with no self-service exit until a manual label nudge cleared it in 2 m 27 s; and (b) rapid re-review churn — 7 bot review events in ~90 s fired 7 CI runs on one commit, 6 were cancelled, and the cancelled runs left the required check permanently unconcluded. The fix list below covers both.
New evidence (2026-08-10, folded in from the pipeline session): the belief that “auto-merge disarms on push” protects us from merging on a stale approval was disproven today — #5625 auto-merged on a verdict from an older commit despite that safety (recorded in the living doc as the D2 counterexample). Stale approvals are now a demonstrated merge path, not a theoretical one, which raises the urgency of this pick's “a green must mean what it says” theme.
Why it ranks #2: it converts the pipeline from “green if a human shepherds it” to “green when it's green.” Combined with pick 1 it removes the mechanism behind most D1 instances (the doc's own analysis: cancellation → stranded pending → nudge).
Some of our traffic lights can get stuck showing nothing at all — not red, not green. Cars queue behind them until a person walks over and kicks the box (we did that about 500 times in three weeks). The fix is boring but total: every light must always end at red or green, on its own, every time.
What: 51% of all PR open-hours happen after the final LGTM — approved, green-ish, idle. The pieces: hold-for-review sits on 11 of 24 open PRs with no owner and no expiry (~72 PR-days lost; #3087 waited 36 days for a “cut day” never scheduled); approved-and-green PRs that never re-arm rot into conflicts within 2–7 days (#5576 collected 12 APPROVED verdicts while BEHIND); session death orphans PRs with no owner of last resort (finished work sat unowned 21h); two sessions racing one PR duplicated whole review laps. The field independently confirms this is the dominant failure: in the largest agent-PR corpus (932k PRs), 38% of failed agent PRs die of reviewer abandonment, not wrongness.
Measured 2026-08-10 (probe): when armed, the machinery is nearly instant — arming → merged in ~1–2 seconds on a green approved PR. The same PR had just sat 6+ minutes doing nothing solely because a stale-branch sync had silently un-armed it. The treadmill (Pick 1) manufactures this idle state on every open PR, every merge — which is why these two picks compound.
Do: (a) the Sweeper — a scheduled job that re-syncs and re-arms any green, non-held, untouched PR (spec already written in the living doc, never built); (b) every hold-for-review gets an owner + expiry, surfaced in a weekly triage note (yesterday's one-off triage of 18 held PRs yielded 4 MERGE / 11 REWORK / 3 CLOSE — make that recurring); (c) a per-PR ownership claim that sessions honor, plus an owner-of-last-resort rule for orphans. Zero new vendors; mostly one workflow + conventions.
Half of all waiting happens after everyone has already said yes. Finished cakes sit on the counter because nobody's job is to box them. The fix is a conveyor at the end of the line: anything approved and untouched gets picked up automatically, anything deliberately held has a name and a date on it, and no cake is ever “not mine.”
What: The August fire. Review rounds per merged PR went ~3 (July) → ~13 (August); #5639's own analysis: “rounds, not findings, are the latency multiplier” — each round is a ~7-min Opus review plus a restarted test lane. Root cause was policy, not model: the rubric said “surface every nit; do not cap,” which is correct at round 1 and an infinite loop at round 7. 60% of post-verdict rework commits touch non-production files. The first fixes landed this week (incremental re-review #5639; round-count guidance after Gera's #5495 note). The field says push further, and how: reviewer precision across the industry runs 16–51%, so every false positive converts into a full round — the fix is a confidence bar, not a better model. Cloudflare holds 131k reviews/month at 1.2 findings per review with an explicit “what NOT to flag” + a judge pass; Anthropic's practice for keeping quality while loosening: every incident's PRs become permanent reviewer eval cases (20–50 tasks from real failures, regression evals held at ~100%).
Why more rounds can't buy quality (measured): our own golden-bug baseline — 8 real shipped bugs replayed against the reviewer — scored 2/8 caught on the old model and 4/8 on the current one, and all four remaining misses were invisible to any diff-reading review: they required a rendered page, a click-through, or production-shaped data. The reviewer's ceiling is the method, not the model — so cutting rounds costs less quality than it appears, and the real recall lever (the rendered-page verification step we built for exactly this) currently runs opt-in and is observed skipping.
Do: (a) confidence-gate the verdict — only high-confidence, high-severity findings block; the rest post as batched non-blocking notes (this is Q6 in the existing review brief, proposed, unimplemented); (b) risk-tier review depth by diff class instead of size alone — a schema/auth/money diff always gets the full treatment, a rename never does; (c) the quality counterweight that makes (a) safe: build the reviewer eval corpus from our real incident PRs (#3521, #2975, the capped false-green class, #5630's life-safety catch) and require it stays at 100% before any loosening ships. Success metric: median rounds ≤2–3 at unchanged catch evidence. (A fourth element — a paid credit-billed deep-review tier for the riskiest diffs — was proposed and rejected by Fede, 2026-08-10: consistent with the standing rule that review runs on the subscription, never metered credits. Do not re-propose.)
Our inspector went from asking for 3 fixes per job in July to 13 in August — and each request stops the whole line, even when it's about a comma. The fix: the inspector still writes everything down, but only serious, near-certain problems stop the line; small stuff ships as one batched note. And before we relax anything, we test the inspector against every real accident we've ever had and prove they'd still catch all of them.
What shipped: The daily 6 p.m. What's Shipping email now carries a lead-time block computed by a single script, merged through full review with 26 tests pinning the definition. The definition is locked: raw open→merged for every merged change — no time filter, no exclusions — median headline, p75/p90 alongside, and share-over-3h promoted to its own headline instead of being filtered away. The two competing filtered definitions that circulated in our own docs were deleted, not deprecated; the abandoned July canary was retired. Why raw won: the old ≤3h filter would have erased the entire August regression, because the regression lives in the tail the filter removes.
First real numbers (Aug 10): median 5 h 30 m over the last 24 h, with 63% of changes taking over 3 h. Green→merged is 30 seconds at the median — the merge machinery is not the wait; getting to green is, across ~17½ reviewer rounds per change. The 30-day median is 31 minutes against 4 h 46 m for the last week: the slowdown is recent and steep, exactly what a filtered number would have hidden.
Honest gaps, stated in the email itself: time-to-recover prints “unavailable” because the incident log hasn't been written to since March — a standing process gap worth closing separately. Change-failure rate is an upper bound from wording, not a verified count.
The factory now has a clock on the wall, read out every evening. It already told us two things on day one: the line got dramatically slower only recently, and the slow part is the back-and-forth with the reviewer — not the final merge step, which takes half a minute.
| Candidate | Status | Why not top-5 |
|---|---|---|
| Re-price #5610 (green review check requires a verdict at that SHA) | FEDE CLOSED 08-10 | Your call stands. Noting only: pick 1 removes ~39% of pushes (the syncs), which shrinks the “per-push review cost” that killed it. Re-openable later with pick 5's number attached; not re-litigated here. |
| Merge-queue retrial with real batching | SETTLED | Trial ran 1 PR/train by construction (ALLGREEN), but the deeper fact holds: arrivals (~12+ min apart) are slower than the suite (8–12 min) — there is nothing to batch. Your own revisit trigger: escalation mass up an order of magnitude. |
| Stacked PRs (GitHub public preview 07-30) | NEVER TRIED | Optimizes human review ergonomics we don't have; squash-merge (our standard) breaks stack identity; best field account calls it “busywork with no upside” for agents. |
| Release train / merge≠deploy decoupling | NEVER TRIED | Adds delivery latency by design — against the goal. Tip-only promotion already coalesces deploys naturally. Revisit if unattended-overnight-merge risk grows. |
| Vercel promote reliability (fails ~daily; 9h-stale-prod class) | DIAGNOSED 08-10 | Fede pulled this into scope; a three-agent investigation root-caused it (see the outcomes rows above: build-machine memory ceiling + 40% of green builds never deploying). Fixes are the deploy decision block below; two are built as held PRs. |
| Auto-merge identity → GitHub App installation token (from human PAT) | NEVER TRIED | Quick win, not a bottleneck: better rate limits, bypass-actor granularity, resolves the reviewer's dual-identity ambiguity. Bundle into pick 2's workstream if adopted. |
| Second reviewer lens (Copilot code review / gh-aw) | NEVER TRIED | Comment-only by design (can't gate); adds rounds while rounds are the fire. Field data (findings barely overlap across tools) makes it interesting after pick 4 lands. |
| Independent-oracle gating tests (a gating test not authored by the session that wrote the code) | NEVER TRIED | The strongest quality idea in the field research (same-author tests are systematically biased); it adds work per PR, so it trades against lead time. Candidate for the dangerous-diff tier only. |
| CI cost moves, preview gating, standards-as-code, subscription-based review | DONE | Field research confirms we're at or ahead of 2026 practice on all four. |
The session that ran the last 48h of pipeline repair handed off today; its open threads are absorbed here so there is one decision surface, not two.
propflow-code-reviewer[bot]) has unverified provenance — verify before trusting its reviews; #5631 still open with auto-merge armed (it will land itself when green).The full measurement changed the priority. The crashing builds (random deaths at the build machine's memory ceiling, 11.3% of deploys) are only 17% of the time prod spends behind. The dominant cause: when merges land close together, each deploy run steps aside for a newer one that also steps aside — 40% of green builds never deployed, prod sat behind for hours with everything green, and a skipped deploy reports "success" so nothing pages. The fix is three parts. Update: fixes 1 and 2 are BUILT — green, bot-reviewed (the reviewer caught two real silent-failure bugs in the fix itself, both corrected and now covered by tests replaying six real deploy timelines), and held for your one-click merge. The build-machine price is now real, from the vendor's own pricing page: doubling the machine's memory (the step that clears the ceiling) runs about $158/month more at current build volume — with one caveat only the dashboard can settle: if our current builds are on the vendor's unbilled default, the real new line item is ~$315/month total, not a delta. Reading that one project setting is a 30-second dashboard check.
Field verdict: straight-to-prod with bounded-blast-radius test fixtures (the Willows pattern) IS 2026 best practice — agent bugs are mostly “context bugs” invisible anywhere but prod-shaped environments, and the field's conclusion is that rollback infrastructure compounds while staging ceremony doesn't. What's missing is not staging; it's three holes in the net that makes straight-to-prod safe: (1) rollback only re-points the website — the background workers that text tenants and write to the PMS keep running the bad code; (2) post-deploy verification is manual and shallow (read-only page loads; the Willows rig is human-invoked — our own golden-bug data says the escaped bugs need a rendered page/real data to see), so the natural fix is a Willows heartbeat: a scripted interaction auto-run after every deploy, paging on behavioral breakage; (3) no deploy-freeze when prod alerts fire post-deploy. Lead-time impact is small; risk impact is large — and Pick 4 (fewer blocking review findings) leans on this net being real.
We don't have a dress rehearsal — every change goes straight on stage. The research says that's fine, even smart, if three things are true: you can pull a bad act off the whole stage (today we can only pull half of it), someone automatically watches the first minute of every new act (today a person has to remember to look), and the show pauses on its own when the audience starts booing (today it doesn't). None of this makes shipping faster; it makes fast shipping safe.
✅ DECIDED — Fede, 2026-08-10: “straight to prod is good for now — don't get distracted on stage.” Straight-to-prod affirmed; prod-leg hardening parked. The three-hole analysis above stays as the record for whenever an incident or a further review-loosening step forces the revisit. No work happens on this tranche.
A private lab copy of the codebase now exists where the real pipeline machinery runs (same CI, same AI reviewer, same merge rules — deploys, Slack, and schedules surgically removed and verified silent). It holds a corpus of 9 real past PRs — 3 fast happy-path, 6 pathological (treadmill victims, a review churner, a stranded-check case, a post-approval rotter, a dead-review case) — with every patch verified to apply cleanly. One command flips the lab between today's rules and an experiment (e.g. treadmill off), so any pick above can be proven on real work before it touches production.
Validation better than planned: the lab's first-ever test PR reproduced two of the diseases live — a stale “changes requested” verdict kept blocking after the reviewer had moved on, and duplicate review-triggered runs cancelled each other leaving the required check stuck red. The lab faithfully has our problems, which is exactly what it's for. Costs: replaying the 3 happy-path samples ≈ 130 CI-minutes per arm; the full corpus ≈ 3,300/arm (won't run without an explicit override — the worst sample alone is 1,141). One item needs Fede: full-fidelity auto-merge arming in the lab needs the merge token pasted in (the builder correctly refused to substitute a broader-scoped one); a native fallback works meanwhile.
Sources: pipeline-delivery.html (the living doc, updated today) + live GitHub API measurements taken 2026-08-10 + git/gh archaeology of ADRs #0075/0077/0079/0107/0120, PRs #2975/#3521/#4468–72/#4983/#4987/#5015/#5518/#5610/#5630/#5639, + four field-research sweeps citing primary vendor docs and 2026 papers (dates inline in the research transcripts). Review-rounds counts (3.0→13.1) are n=50/window samples, directional. The ≈44h/week treadmill figure is an estimate (syncs × mean lap), labeled as such. One workstream — session-transcript archaeology (the deeper story behind Fede's past pipeline decisions) — was still running at publish time; findings that change any ranking will ship as a revision to this page.
Constraint honored throughout: nothing here touches the dangerous-diff hard floor, the review gate's existence, or arms/sends. Pick 4 explicitly trades noise for speed, never catches — its guardrail (incident eval corpus at 100%) is the mechanism that makes that claim testable rather than hopeful.