# Proposal 2: reliability first (zero red main)
Labels: FACT = from the findings files / red-main logs; EST = estimate, basis given.

## Why main goes red (FACT, red/summ.txt: 83 failed main job logs, 1,000 recent push:main CI runs)
| Cause class | Evidence | Share of failing test lines | Mechanism that kills it |
|---|---|---|---|
| Drift pins broken by two PRs that were each green alone (semantic merge conflict) | user-route-inventory 191, property-scope registry 10, specialist-prompt/transfers 13, identifier fence 1 | ~69% (215 of 313) | Merge queue tests PR + latest main before merge; drift set always runs in queue (never "affected"-skipped) |
| Repeat-failing single test (flaky or real, unowned) | maintenance-comms-activities 85, marketing-home-hero-fit 9 | ~30% | Flake detector: fails then passes on same SHA -> quarantine lane + owner issue; real -> auto-revert |
| Runner never launched (VcpuLimitExceeded, 320 vCPU) | 3 of 11 read PR failures (ci-gates-inventory §4) | ~1 in 6 failed runs | Auto-retry on launch failure only (not on test failure), max 2, then spot->on-demand pool |
| Behavior bug CI cannot see | Sep 16 empty post_dial_digits killed every call; 17 reverts/30d | n/a (prod) | Prod canary + auto-rollback per lane |
Today: 202 failure of 1,000 push:main CI runs (20%), 607 cancelled (coalescing, by design).

## Target pipeline
```mermaid
flowchart LR
  A[pre-push local<br/>review:self, typecheck, affected unit] --> B[PR: fast gate<br/>lint, types, affected unit, drift set, nested, review bot]
  B --> C[merge queue<br/>batch 2-4, full suite once vs latest main]
  C --> D[main: deploy tip<br/>web, lambdas, worker]
  D --> E[prod verify<br/>10% canary 5 min, probe + error rate]
  E -->|breach| F[auto-rollback + auto-revert PR]
```
| Stage | Runs where | What | Wall (EST basis) |
|---|---|---|---|
| Pre-push | local | review:self, typecheck:scripts when scripts/ touched | 2-6 min (FACT: pre-push ~6 min) |
| PR | RunsOn spot, 1 workflow | lint, types, build, affected unit, **always-on drift set** (*.drift.test.ts), nested lane, Claude review (🔴 = correctness only) | ~8-11 min (FACT CI wall median 11.2) |
| Merge queue | RunsOn spot, `merge_group` | full unit 4 shards, guard pins, leasing regression, build; batch 2-4; launch-fail auto-retry | ~11 min |
| Post-merge | deploy controller | deploy main tip to every lane (no pinned SHA), no main CI rerun | ~5-8 min (Vercel build 4.4 FACT) |
| Prod verify | Vercel + Lambda alias + ECS | 10% traffic 5 min, Sentry error rate, synthetic call/sms/login probe; ECS circuit breaker; freshness alarm 45 min | 5 min |

## Gates: delete / fold / keep
| Gate | Action |
|---|---|
| Push:main full CI rerun (181/day, 60% cancelled) | Delete (queue already tested the exact merge result) |
| 30-min promote batching | Delete (kept only as cold-start warm step) |
| Claude Code (@claude), Docs-only fence, Dependabot gate, Dispatch Guard, Branch Preview per push | Delete from push path (FACT: 0 work on ~12 of 22 runs/push) |
| Override, Recheck, Re-pin, infra-rerun, verdict watchdog | Fold into claude-code-review.yml |
| Nested Package CI, PR body gate, size tier, self-review, Quality Gate POC, Semgrep | Fold into CI as lanes (Semgrep to nightly) |
| auto-merge-all.yml rules "base moved", "red-main freeze", quiescence | Fold into merge queue (queue owns ordering) |
| Build, Type Check, Unit Tests, Lint, Regression Gate, review | Keep (required) |
| Dangerous-diff hold | Keep |

## Numbers
| Metric | Today | Target | Basis |
|---|---|---|---|
| Runs per push | 22 (FACT) | ~3 (CI, review, queue) | fold table |
| Machine-min per PR | ~176 (55/push x 3.2 pushes, FACT) + ~72 review (13k min/day / ~180 PRs) | ~75 | PR fast gate ~12 x 3.2 + queue 43/3 per batch + deploy 5 (EST); review extra |
| $/month compute | ~$750 CI + ~$325 review (EST: 5,437 PRs x min x $0.05/2-vCPU-h) | ~$340 CI + review unchanged unless tiered | spot $0.05/h; on-demand fallback only on launch failure |
| Claude review tokens | not logged (FACT) | log per run | - |
| PR -> prod median | ~0.36 h merge + 18-49 min web; worker/lambda hours on bursts | ~50 min (0.36 h + 11 queue + 8 deploy + 5 canary) | EST |
| Red main failures | 20% of push:main runs | < 3% | drift 69% + flaky ~30% + infra removed; residual = queue-batch misses |
| Red-main reduction | - | ~85% (EST) | per class above |
| Merges/day capacity | ~181/day today | ~800/day | queue: 11 min, batch 3, concurrency 4 -> ~65/h x 12 busy h (EST) |

## Migration (8 PRs)
1. Deploy controller: deploy main tip for lambdas (finish #14289 pattern), freshness alarm 120 -> 45 min.
2. Launch-failure auto-retry (VcpuLimitExceeded only, max 2, then on-demand pool) + one shared RunsOn spec (163 copies -> 1).
3. Always-on drift set lane in PR CI (never skipped by affected-tests).
4. Flake detector + quarantine lane: same-SHA fail-then-pass -> quarantined, owner issue, 7-day fix-or-delete.
5. Add `merge_group` trigger to CI; enable GitHub merge queue (batch 2-4, concurrency 4); auto-merge guard hands ordering to queue.
6. Delete push:main CI rerun and promote batching; fold 5 review helper workflows; delete 7 no-op satellites.
7. Prod canary: Vercel staged 10% + probe + Sentry rate with instant rollback; weighted alias canary for 7 alias lambdas; ECS circuit breaker for worker.
8. Auto-revert: canary breach or queue-bypass red main opens a Revert PR and merges it ([urgent]), posts #alerts.

## Risks
| Risk | Mitigation |
|---|---|
| Queue batch failure costs a bisect and delays 2-4 PRs | small batches; queue drops the failing PR and re-runs the rest |
| Merge queue changes the auto-merge guard (dangerous-diff floor) | attended merge after full-diff read; guard keeps its dangerous-diff hold |
| Canary probes place real calls/texts | probe only the Willows test property |
| Quarantine becomes a graveyard | 7-day fix-or-delete, count on nightly report |
| vCPU cap still binds at peak | fewer runs per push frees capacity; spot pool diversification, no GitHub-hosted runners |
| Removing push:main CI loses a safety net | queue tests exact merge commit; prod canary + nightly full run as backstop |