CI/CD factory-floor map

This section is synthesis. It combines the verified numbers in §2–§11, and every inference is marked as one. It does not recommend anything.

The constraint

• Why per-entry failure probability is high (V, §5, §10): PR checks don't run what the queue runs. A PR runs affected tests only (capped at 200 files), a reduced guard-pin subset, and no npm run build. The queue runs the full guard list, the 64-core full suite, and the build. So the queue is the first place many tests ever execute against a given change. In 18 of 22 large red-main incidents the culprit PR had a green Unit Tests, and its Guard pins were usually skipped.
• Why a failure costs so much (V): the chain is deep (24 entries), a run takes 14–29 min (median 17), and a failure invalidates everything behind it. A rebuilt group pays full price again: the only reuse is the tree-hash skip on push:main (§5.5), and nothing reuses results inside the queue.
• Arithmetic, I (not measured): about 21% of non-manual queue exits are failures (12% failed checks + 9% text conflict, §10.4). If each entry fails independently at even 10%, a 24-deep chain is fully green only about 8% of the time (0.9²⁴). Most speculation is therefore built to be thrown away. Throughput is set by how often the chain stays green, not by how many machines exist.

Upstream inventory: where work piles up

• The queue itself (V): at 22:00Z it held 24 entries, exactly the build cap, and the head entry had waited 32 min. Queue dwell is median 35 min and p90 155 min (proxy, §9.5).
• Open PRs (V): median 141 min from PR open to merge (p90 230). The median PR lands about 28 commits after its base (p90 58).
• Re-entries (V): 990 queue entries for 719 PRs. 168 were manual removals, 94 of them a bulk dequeue on 10-02 between 19:00 and 21:00 by an actor that is U.

Downstream starvation

• Deployment is not starved and is not the constraint (V/I). Production promotes in deliberate 30-min batches: 39 deploys in 18.5 h, median 5 commits each. Main to live is a median 53 min. That is extra latency layered on top, but it is not what limits safe merges.
• Red main does starve deploys (V): a red push:main run blocks the web promote, Lambdas and the worker together, because they are one gated unit (§8.0). Before the queue, 644 failed main runs since 09-18; multi-run red-main incidents lasted a median 0.5 h and p90 4.2 h.

Scrap and rework

• About 1,500 cancelled queue runs a day, at about 56 exec-min each. That extrapolates to about 86,000 exec-min a day of discarded queue work (I, n=15 sample, §11.1).
• The full suite runs twice inside every queue run (6 guard-pin shards plus the 64-core full set), again every 30 min on main (post-merge-heavy), and the guard list runs a third time inside the full set (V, §11.2).
• About 80% of push:main runs end cancelled. These are superseded promote jobs, not discarded tests (V, §11.5), so this is cheap scrap.
• Every rebase or merge of main costs a full LLM review round, because clean verdicts don't carry forward (V, §9.4). Rebasing is discouraged by doctrine, so this cost mostly lands on CONFLICTING PRs (I).
• Red-main fix PRs: 73 since 09-18. 28 of them fix drift in pinned or generated artifacts (counts, line numbers, sidecars, inventories) (V titles, I grouping).

Variability: what makes throughput collapse

• A test that never ran before merge: one bad entry reddens the whole chain. #14896 alone produced 47 failed queue runs over about 2 h (V).
• Shared-list and count ratchets: one guard test failing across many unrelated PRs. sync-tour-mock-covers-both-views failed 33 runs across 19 PRs in about 2.5 h (V).
• Changes to CI itself: about 20 ci.yml merges between 09-30 and 10-02, including 2 reverts. Some of them changed which tests the queue runs (V).
• Capacity events: on merge_group, runner start latency p90 was 264–315 s (max 1,308 s) during a burst around 20:05–20:19Z (V). Separately, AWS VcpuLimitExceeded (5 red-main incidents), a full runner disk (11 of 171 sampled cancellations), and spot reclaims (exit 137) (V).
• The queue switching on and off (V): it was off 10-01 20:00–23:00, when 84 PRs merged directly in one hour and main went red 45 times.

Utilization

Yes: agent utilization is high while system throughput is set elsewhere (I). Agents produce PRs faster than the chain can absorb them at the current per-entry failure rate. Each extra queued PR deepens the chain, and a deeper chain raises the odds that something ahead fails and forces a rebuild. Beyond some depth, more PRs in flight means more scrap, not more merges. Where that depth sits is not measured (see §13).

Which category: A–H

  AGENT SWARM  (~200–470 PRs/day landed; worktree off fresh origin/main, enforced by hook)
      │  pre-commit: regenerate + stage route manifest   pre-push: symbol/docs-index/... checks, push-pacer
      ▼
  PR opened ─────────────────────────────────────────────────────────────────────────────
      │  pull_request → ci.yml on refs/pull/N/merge  (= PR + main AS OF THIS PUSH, never refreshed)
      │    Type Check (tsgo, whole repo) · Lint (whole repo) · Unit Tests (AFFECTED, cap 200)
      │    Guard pins (SUBSET, 3 shards) · always-run drift/decide-* · Leasing gate (path-gated)
      │    Build: NOT RUN (left to Vercel) · evals: path-gated, PR head only
      │  claude-code-review (LLM, PR commits only; clean verdict does NOT survive a new SHA)
      │  branch protection strict=false → being behind main is allowed
      ▼
  ENQUEUE  (auto-merge-all.yml DISABLED since 10-01; enqueued as gera-/fede-propflow — actor U)
      ▼
  GITHUB MERGE QUEUE   ruleset: ALLGREEN · build ≤24 · merge ≤16 · wait 2 min · timeout 30 min · SQUASH
      │
      │   base M ─▶ [M+A] ─▶ [M+A+B] ─▶ [M+A+B+C] ─▶ ... up to 24 deep, all running at once
      │               │          │            │
      │   each group runs ci.yml on merge_group (17 min median):
      │     Checks (lint + scripts typecheck + npm run build) · Unit Tests (affected vs base_sha)
      │     Guard pins (FULL, 6 shards) + Full set (64-core: full guards + full suite) ← suite runs 2×
      │     review = PASS-BY-DESIGN · Leasing gate = PASS-BY-DESIGN · Queue fail-fast watcher
      │
      │   B red ─▶ fail-fast CANCELS B's run (reads "cancelled", not "failure")
      │          ─▶ every group built on B is red/void ─▶ GitHub ejects B, REBUILDS C, D, ... on M+A
      │   ~4.8 queue runs per merged PR · 68% of runs end cancelled · text conflict = 9% of exits
      ▼
  MAIN  (squash; ~19 commits/hr; up to 16 PRs per push)
      │  push → ci.yml: skips tests whose tree the queue already validated; checks only
      │  post-merge-heavy (cron :07/:37): full suite on tip → culprit-find, auto-revert, hold deploys
      │  sync-specialists / sync-phone / turnover-intake / DLQ alarms ── fire on raw push:main,
      │      NOT gated on CI ─▶ ElevenLabs + CloudWatch live within minutes
      ▼
  ONE GATED RELEASE  (needs push CI = success)
      ├─▶ vercel-promote: waits a 30-min batch window, promotes the newest green tip ─▶ propflowai.co
      │       (one runs, one waits, a newer arrival replaces the waiter → ~80% of push runs "cancelled")
      ├─▶ deploy-lambdas: esbuild import-graph gate per lane (fail-open) ─▶ 13–15 of 15 lanes usually
      │       ─▶ zip ─▶ aws lambda update + alias live          (6–8 min after CI)
      └─▶ deploy-renewal-worker: source hash over src/agents/... ─▶ arm64 image ─▶ ECR ─▶ ECS Fargate
              (almost every merge rebuilds it; 13–34 min)
      ▼
  PRODUCTION   web lags main by median 53 min (p90 90) · one DynamoDB table shared by every unit

  RUNNERS under all of it: RunsOn on EC2 (3 stacks: us-east-1 2-AZ "production", us-west-2, us-east-1 6-AZ flex),
  one instance per job, spot + on-demand, GitHub Actions cache (10.68 GB of 10 GB cap), no Docker/K8s
  except the worker image build.

The section reports were each gathered by an independent read-only pass. Where two passes measured the same thing over different windows, the numbers differ slightly. Examples: merge_group cancellations are 1,539 in the trailing 24 h vs 1,377 on the calendar day 10-02; required checks are 6 from classic branch protection plus Nested Gate from the ruleset.

PropFlow CI/CD forensic map: §1 Topology, §6 Dependency graph, §7 Hotspots

Pinned to origin/main @ 864d9a57570. Tags: V = VERIFIED (cited), I = INFERRED, U = UNKNOWN.

§1 Repository topology

Shape: a single monorepo, a single TypeScript program, a single root lockfile, and no monorepo build tool. (V) There are 22,015 tracked files (git ls-tree -r). No turbo.json, nx.json, Bazel, cdk.json, SAM template.yml or serverless.yml appears in the tree. None of those tools appears in root dependencies either; the positive control is that next 16.3.0, esbuild, vitest and @typescript/native-preview (tsgo) all do appear. Package manager: npm ≥11, node ≥24 (package.json engines). Root package.json has 482 npm scripts, 76 dependencies and 27 devDependencies.

• **npm workspaces: packages/*, which contains exactly one package, @propflow/ui. (V)** 544 src files import it.
• Nested packages outside the workspace (V): agents/clara (@propflowai/agents-clara, own package-lock.json), mock-pms-service (own lockfile, vercel.json and next.config.js, so it is a separately deployable Vercel app), quality-gate-poc (own lockfile), testing-harness, tools/graphify. There are 4 lockfiles in total.
• Languages (V): TypeScript/TSX (Next.js 16 App Router, React 19), Bash (deploy scripts, hooks), Terraform (infra/runs-on-flex/*.tf, which defines the CI runners), and Python in pockets (evals/fast-tier, infra/temporal-worker/preflight-secrets.py).

src/ (12,748)        Next.js app on Vercel: app/ (528 route.ts, 198 page.tsx), lib/, components/, __tests__/
  lib/data/types.ts  22,852 lines, the canonical entity schema (no codegen; hand-written + JSDoc @canonical/@derived)
agents/clara/ (655)  Clara agent runtime; lib/ mirrors src/lib (parallel data/types.ts, 3,561 lines)
lambda/ (200)        17 function dirs, each with build.ts (esbuild) + deploy.sh; lib/esbuild-clara-fallback.ts
services/renewal-worker/  the ONLY Dockerfile; Temporal worker on ECS Fargate
infra/               runs-on-flex (Terraform: CI runners) · temporal-worker (shell + IAM/task-def JSON, no IaC framework)
packages/ui (36)     the one workspace
scripts/ (4,153)     ops tools, DDB migrations/GSI backfills, CI helpers, harnesses (separate tsconfig.scripts.json)
evals/ (1,732) · e2e/ (59) · tests/ (11) · evidence/ (262) · quality-gate-poc/ (210)
config/ (13)         phone-registry.json, elevenlabs-agents.json, automations.toml …
contracts/           1 file (change-term/scenarios.json), NOT an API-contract layer
elevenlabs/          1 file (.gitkeep)
docs/ (1,343)        INDEX.md generated by scripts/generate-docs-index.ts; 169 entries in docs/adr/
.github/ (84)        64 workflows; CODEOWNERS
.husky/              pre-commit, pre-push, post-checkout
root clutter         .pr-body-*.md, PROGRESS-*.md, ITERATION-2-PLAN.md, chunk-complete-draft.md, e2e-rent-roll-test.mjs

Schemas and migrations (V): the schema is code, src/lib/data/types.ts plus a parallel agents/clara/lib/data/types.ts that describes itself as a "PARALLEL TREE" (header lines 1–12). There is no migrations directory. DynamoDB GSI changes are one-off npm scripts (gsi::add, gsi::backfill, gsi::revert, parity: in package.json).

Generated, committed files (V): src/lib/traffic/route-manifest.generated.ts and coverage.generated.ts (regenerated and staged by .husky/pre-commit), plus docs/INDEX.md (npm run docs:index).

Coupling (V): the boundary between agents and src exists only on paper.

• Root tsconfig.json maps "@/lib/": ["./src/lib/", "./agents/clara/lib/*"], includes **/*.ts, and has no references. Only scripts, evals/fast-tier, quality-gate-poc and quality-replay are excluded. So src, agents, lambda, packages and e2e typecheck as one program. The ci.yml commit "tsc heap cap 12288 → 20480MB (#15374)" is a symptom of that.
• The ESLint CROSS_RUNTIME_PATTERNS (eslint.config.mjs:8-30) blocks only literal src/ and agents/ paths. Its own comment says the alias fallback still bridges the two trees.
• I resolved every @/lib/X import against the tree with the importer-first rule that lambda/lib/esbuild-clara-fallback.ts uses:
  • 1,114 import statements in agents/ resolve into src/lib (451 distinct targets).
  • 874 statements in src, lambda and services resolve into agents/clara/lib (182 targets).
  • 59 relative paths exist in both trees (for example data/dynamo/index.ts and data/dynamo/conversation.ts), and which one is used depends on who imports it.
• Literal cross-tree imports: lambda → src/ 27 files, lambda → agents/ 3, scripts → src 1,264, scripts → agents 113, evals → src/agents 219.
• services/renewal-worker/index.ts imports @/lib/temporal/workers/* directly.

Can each area be built or deployed on its own?

• Separate: mock-pms-service (own app), infra/runs-on-flex, packages/ui (has its own build.ts). (V/I)
• Every Lambda, the renewal worker and the Vercel app bundle from the same src + agents source. They deploy separately but are coupled at build time. (V)

§6 Dependency graph: "if X changes, what rebuilds or retests?"

What tooling exists

Four kinds of dependency

• File overlap (V): see §7.
• Build dependency (V): one tsc program, plus the alias fallback across both trees. Changing src/lib/data/types.ts (4,025 direct importers via @/lib/data/types) re-typechecks everything. Changing it also, by construction, changes the input of every esbuild bundle that reaches it.
• Runtime dependency (V): a single-table DynamoDB design. Both src/lib/data/dynamo-client.ts:239 and agents/clara/lib/data/dynamo-client.ts:70 read TABLE_NAME = process.env.DYNAMODB_TABLE_NAME || 'propflow-data'. Lambdas pin propflow-prod / propflow-stage (lambda/appfolio-sync/deploy.sh:75-76), and so does the ECS task definition (task-definition.template.json:28). The two type trees describe the same rows. A row-shape change in one tree compiles cleanly and breaks the other at runtime. (I)
• Deployment dependency (V): one merge can deploy Vercel, up to 16 Lambdas and the ECS worker. Lambdas wait for workflow_run on a green push:main CI run; the worker uses a source hash.

Concrete examples

• A. Different files, genuinely independent: infra/runs-on-flex/vpc.tf against any src/app/**/page.tsx. (I; this is a negative and is not proven.)
• B. Different files, still dependent: adding one data method touches five files: src/lib/data/repository.ts (IDataRepository), src/lib/data/index.ts, src/lib/data/dynamo/index.ts, src/lib/data/json-repository.ts and src/lib/data/store.ts (8,087 lines). Two PRs that each add a method touch the same five files. (V for the structure; I that every method needs all five.)
• C. Textual conflict on a central file: src/lib/data/types.ts (419 commits in 30 days), package.json (203), .github/workflows/ci.yml (169). (V)
• D. Merges cleanly but breaks semantically:
  • (V) The header of route-manifest.generated.ts records that two PRs each changed a count line from N to N+1. Git merged the identical edits, main landed one short, and push:main went red (#9921, #9948, 2026-09-19). The file now stores no counts, and docs/INDEX.md does the same.
  • (I) eslint-suppressions.json (192 files, 765 suppressions): one PR removes file X's entry after fixing X, while a concurrent PR adds a new violation in X. Both merge cleanly and lint goes red.
  • (I) A new ratchet test pins a count while a parallel PR moves that count.
  • Size of the D class (V): 665 .drift.test.ts files (7,931 test files in total). Of those, 221 mention allowlist, ratchet, baseline, ceiling or grandfather, and 210 walk the tree (readdirSync, glob, ls-files). 129 tests read .github/workflows. 32 drift tests parse types.ts. There are 71 allowlist/baseline/ratchet files outside docs/. These tests turn any concurrent tree-wide invariant into a semantic conflict. The CI agent should look at "PR lane always runs drift and decide- guard tests (#15617)".

§7 Change and conflict hotspots (last 30 days)

Throughput (V): 6,512 commits on main, 6,467 distinct (#N) PR refs, only 20 true merge commits (squash merging), and 45 commits with no PR ref. That is about 209 PRs/day; median 175/day, max 542/day. The brief's "~100/day" is I and looks low. Authors: "Gera" 3,845, "fede-propflow" 2,619.

Top serialization points

Also notable (V): scripts/operator-rename/ratchet.json 45, config/phone-registry.json 30, eslint-suppressions.json 23, customer-identifier-fixture-baseline.ts 20, agents/clara/lib/data/types.ts 72 (the parallel schema), and claude-code-review.yml 60 and auto-merge-all.yml 52 (the merge machinery is itself a hotspot).

docs/INDEX.md is enforced but cool: 53 changes (V). It is enforced by the pre-push check-docs-index.ts, by the docs-index step in ci.yml and by a full rebuild in ci-docs.yml. It is lower churn than the brief implied.

Hooks that force edits to central files (V)

• .husky/pre-commit regenerates and git adds the route manifest and coverage files whenever src/app/**/(route|page).ts(x) or src/lib/traffic/surface.ts changes.
• .husky/pre-push runs check-symbol-sweep, check-mutation-control, check-deleted-file-importers, check-elevenlabs-deploy-lane, check-docs-index (diff-scoped against $BASE) and check-husky-runtime.

Pattern (I): most hotspots are registries and inventories that a drift test keeps closed: the types classification, route inventories, scripts baseline, key registry and the generated manifest. Each turns "add one thing" into "edit one shared list". Lists merge cleanly in Git, but count lines and ratchet ceilings collide semantically, as in example D.

PropFlow CI/CD forensic map: §2 PR lifecycle, §3 merge queue

Pinned to origin/main @ 864d9a5757 (git rev-parse origin/main). Live API snapshot taken 2026-10-02 at about 21:58–22:00Z. Tags: V = verified (file:line or command output), I = inferred, I-gh = inferred from GitHub's documented semantics, U = unknown.

§3 How the GitHub merge queue is set up and behaves (most important)

3.1 Configuration (V)

gh api repos/.../rulesets returns one ruleset, 14925316 "Require CI". It is active, targets ~DEFAULT_BRANCH, has bypass_actors: [], and its updated_at is 2026-10-02T13:44:15-05:00, earlier today. The parameters below are current as of the snapshot and may have changed recently.

merge_queue: merge_method SQUASH, max_entries_to_build 24, min_entries_to_merge 1,
  max_entries_to_merge 16, min_entries_to_merge_wait_minutes 2,
  grouping_strategy ALLGREEN, check_response_timeout_minutes 30
required_status_checks: [{context:"Nested Gate", integration_id:15368}],
  strict_required_status_checks_policy:false
pull_request: required_approving_review_count 0, dismiss_stale_reviews_on_push false,
  (other review params off); also deletion, non_fast_forward

GraphQL mergeQueue.configuration returns the same values: checkResponseTimeout 1800s, mergingStrategy ALLGREEN, minimumEntriesToMergeWaitTime 120s. (V)

branches/main/protection does not return 404. Classic protection is still active alongside the ruleset (V): required_status_checks.strict:false, contexts = Build, Type Check, Unit Tests, "Regression Gate — Leasing (required)", Lint, review, all app_id 15368 (GitHub Actions). enforce_admins:false, required_conversation_resolution:false. No required human review.

The effective required set is the union of seven contexts: the six classic ones plus Nested Gate. Neither layer requires the branch to be up to date (I-gh: GitHub layers rulesets and classic protection, and both apply).

3.2 Which workflows fire on merge_group (V)

git grep merge_group -- .github/workflows: only ci.yml has a merge_group: types:[checks_requested] trigger (ci.yml:196). claude-code-review.yml:110 says its merge_group trigger was deleted. auto-merge-all.yml and ci-followups.yml use branches-ignore: ['gh-readonly-queue/**'] on their workflow_run legs.

Per ci.yml:189-195, only the "CORE lane" has a merge_group leg. The jobs that actually run are (V, from the if: conditions):

Which SHA the queue tests: github.sha is the head of the temporary branch gh-readonly-queue/main/pr-<N>-<base_sha> (I-gh). GraphQL confirms it is a chain (V): each entry's baseCommit is the previous entry's headCommit, and entry 1's base is the main tip 864d9a57. Each head is the squash of the PR on top of the commit before it. The <sha> in the branch name is that base commit, so the branch name does not show how many PRs the group contains. Every observed run name is a single pr-<N>.

3.3 Required checks to enter the queue vs. to leave it

To enter (PR head): all seven contexts must be green on the PR head SHA (I-gh: the queue only accepts a PR whose required checks pass). Because strict:false, those results come from refs/pull/N/merge as it stood at the last push, and they can be stale relative to main.

To leave (merge-group SHA): the same seven names must report success on the merge-group commit. In ci.yml they come from three different sources (V):

This is the stub pattern you asked about, and the repo names it openly. The bot review and the leasing regression gate are never evaluated against the combined tree.

merge-group-contexts and merge-group-by-design are named by expression (… && matrix.name || '<name> (not applicable)') so that a skipped job never posts under a required name (#9119). merge-group-verdict breaks that rule: its name is fixed as name: Nested Gate. So every PR-event ci.yml run leaves a skipped Nested Gate check-run on the PR head (V: PR run 37069793874's job list shows Nested Gate skipped). Whether that shadows the real PR-side result is the aggregation question the repo itself marks unsettled (U).

3.4 Live queue activity (V, read-only GETs)

• Queue depth at 22:00Z: 24 entries, equal to max_entries_to_build. States: AWAITING_CHECKS ×15, UNMERGEABLE ×8 (2 have headCommit:null, probably conflicts, I), MERGEABLE ×1. Entry 1 (#15725) was enqueued 21:28 and was still AWAITING_CHECKS at 22:00 (32 min) on its second build.
• Who enqueued: every enqueuer.login is fede-propflow (9) or gera-propflow (15), which are human credentials, not an App (V). The repo credits "an external loop" (auto-merge-all.yml:420, 488). Where that loop lives is U: it is not in propflowai scripts/ or .github/, and agentflow SYSTEMS.md lists only a reader (ci-factory-mq).
• Today's merge_group runs of ci.yml: 2,073 total, of which success 676, cancelled 1,382, failure 0 (V, filtered by status=). Main received 390 commits since 00:00Z (V, commits API).
• "Cancelled" means a real failure, not invalidation. I sampled 4 of the 37 cancelled runs and all 4 had a real red job (Guard pins failing guard-receiver-dot-blindness.drift.test.ts or decide-unit-tests-conclusion.test.ts:983, or Full set red), and queue-fail-fast then cancelled its own run (V). So a queue failure shows up as cancelled, never as failure.
• Concurrent groups: 20 runs were in progress at the snapshot, many started within the same 2 seconds (21:52:09–21:52:19) (V). The concurrency key is ci-CI-merge_group-code-${github.ref} with cancel-in-progress: true (ci.yml:331-333). github.ref is the unique queue branch, so one group never cancels another (V from the expression).
• Timings: successful merge_group runs took min 14, p50 17, max 29 min (n=43) (V). The max is close to the 30-min check_response_timeout.
• A success can still be rebuilt: #15628 went green at 20:49 and 20:50, then was rebuilt and went green again at 21:09. #15725's first run (on top of #15606) was cancelled after #15606's Full set failed at 21:40, then rebuilt on main tip 864d9a57. #15606 re-entered later (V).
• One failure spreads down the queue: 21:47–21:58 saw a cluster of entries fail the same guard-pin test (V, sampled). This fits one bad upstream entry being included in every group built after it (I).

3.5 What happens when an entry fails (I-gh unless marked)

1. A required context goes red on entry k's merge-group SHA, or nothing reports within 30 min. GitHub removes k from the queue.
2. Every entry after k was built on top of k, so GitHub rebuilds those entries on a new chain without k and fires a new merge_group checks_requested for each one.
3. GitHub does not cancel the old runs for groups it invalidated. In this repo they are stopped only by queue-fail-fast when they turn red themselves (V). Old runs that would have passed simply finish and are ignored. Concurrency cannot cancel them because each one has a unique ref (V).
4. ALLGREEN: an entry merges only when it and everything ahead of it are green. Up to 16 ready entries can merge together after the 2-min wait (I-gh). Two same-timestamp commit pairs on main weakly support batching. The gap between 390 commits and 245 push:main runs is confounded by docs paths-ignore, so it is not evidence.
5. A removed PR leaves the queue. It re-enters only if something re-enqueues it. Here that is presumably the external loop, since the enqueuer identities are human (I).

History (V, ci.yml:229-245): the queue ran 07-14 to 07-21, was retired ("NEVER batched (66 trains for 63 PRs)"), and came back 2026-10-01 10:49Z (auto-merge-all.yml:420).

3.6 State diagram

PR head (refs/pull/N/merge at push time, strict:false)
  │  7 contexts green: Build, Type Check, Lint, Unit Tests, Regression Gate,
  │                    review, Nested Gate(nested pkgs)
  ▼
[enqueue] ── by fede-/gera-propflow credential (loop location U)
  ▼
QUEUED ──► build group k = squash(PR) on head(k-1)   [≤24 groups built at once]
  ▼  merge_group:checks_requested → ci.yml core lane (one run per entry, own ref)
AWAITING_CHECKS ──(any verdict job red)──► queue-fail-fast cancels own run
  │                                         → Nested Gate/contexts post failure
  │                                         → UNMERGEABLE → REMOVED
  │                                         → every entry behind k REBUILT on new chain
  │──(no report in 30 min)──► REMOVED (I-gh)
  │──(conflict)──► UNMERGEABLE (headCommit null)
  ▼ all 7 green on group SHA (4 real, Nested Gate=core verdict, 2 by-design)
MERGEABLE ──(all ahead green; ≤2 min wait; ≤16 per merge)──► squash to main
  ▼
push:main ci.yml (mg_validated skips tests) → promote window → deploys

§2 The PR lifecycle, stage by stage

2.0 Workflow enable/disable state (V, a finding that cuts across stages)

15 workflows are disabled_manually (actions/workflows), including auto-merge-all.yml (since 2026-10-01 20:48Z; its last run was 20:48:16Z), claude-review-recheck.yml, claude.yml (@claude review), and the standalone semgrep, docs-only-fence, pr-size-tier and pr-body-gate. The callees still run through ci.yml's uses: (sampled PR runs: Semgrep SAST / Scan=success, PR gates / PR gates=success). auto-merge-all no longer runs at all, so its arm call (auto-merge-all.yml:1082) is moot. Enqueue happens through human credentials (§3.4).

2.1 Per-stage matrix

2.2 Key points from §2

• The bot review re-runs on every push (synchronize, V) and never on merge_group (V). In the queue its context is a by-design pass (§3.3).
• Three revisions get tested and none of them is the next one: the PR lane tests refs/pull/N/merge at push time, the review reads the PR head, and the queue tests a chained group SHA. Only the queue's result is tied to what lands. On push:main the tree-equality check then skips the test lanes (V).
• The full suite runs twice and never per PR: once in the queue (required) and every 30 min on main. The queue skips the ElevenLabs live-drift tests on purpose.
• The documentation contradicts the config (V): docs/runbooks/ci-and-deploy.md:69 says "no merge queue, ADR-0107". claude-code-review.yml:110 says the stub was deleted, but it lives in ci.yml merge-group-by-design. The ci.yml preflight comment lists only 4 of the 7 required contexts. auto-merge-all.yml describes a guard that is disabled.

PropFlow CI/CD forensic map: §4 Execution Architecture and §11 Queue Economics

Pinned to origin/main @ 864d9a57570 (64 workflows). Live data captured 2026-10-02 ~22:05Z. Tags: V = VERIFIED (file:line or command output), I = INFERRED, U = UNKNOWN.

§4 CI execution architecture

4.1 Runner fabric: RunsOn (self-hosted EC2 autoscaler) in AWS account 194925971150, three stacks

• V: CFN stacks from list-stacks/describe-stacks. Flex has no CFN entry; it appears as ECS cluster runs-on-flex plus tagged resources.
• V: infra/runs-on-flex/README.md says the flex stack was "applied 2026-09-30" and "cutover of the heavy jobs is a later PR". Today the old runs-on stack is still Environment=production, so it has not been cut over. I: the 2-AZ stack the README meant to replace still serves every label without an env=.
• V: both CFN stacks: RunnerMaxRuntime=720, control plane on Fargate, AppBudgetDailyUsd=5 (control plane only, not runners), Private=false (no NAT), EncryptEbs=false, EnableCacheIsolation=false, config extends from .github-private.
• V, drift: the spot circuit breaker has three different values. Both CFN stacks run 100/15/5. The flex Terraform sets 20/15/10. The README says 20/15/10 was "carried over as live", which the live CFN stacks contradict.
• V: EC2 quotas (Standard family vCPUs). us-east-1: spot 4096, on-demand 2048. us-west-2: spot 1024, on-demand 1024. I: these quotas are the real ceiling on concurrency, since nothing in the config caps it.
• V: org runners/variables APIs return 403 (no admin:org). The repo runners endpoint lists 530 mostly-offline runs-on--i-<instance> registrations. I: stale per-job registrations, not a fleet.

4.2 Labels and routing

V: no .github/runs-on.yml in the repo; pools live in .github-private:.github/runs-on.yml (last change 2026-10-02T19:06Z, "keep 6 hot"). Nearly every job uses: ${{ vars.CI_RUNS_ON == '1' && format('runs-on={0}/...', github.run_id) || '<GitHub-hosted fallback>' }} V repo variables: CI_RUNS_ON=1 (set 2026-09-28) and CI_CAPACITY=west-only (set 2026-10-02T19:57Z). scripts/lib/no-hosted-runners.ts fails any hard-coded hosted label.

V EC2 snapshot (~1 h of churn, not a day): runs-on-flex: 2,536 t4g.medium (1,911 spot, 625 on-demand), 331+29 m7g.xlarge, 198 m8g.xlarge. runs-on: 522 t4g.small, 447 m7i-flex.large, 212 m6a.large, 48 c7g.16xlarge + 24 c8g.16xlarge (the 64-vCPU full-set). runs-on-usw2: 133 on-demand m7g.2xlarge plus r8gd/m8g/m8gd spot. Anomaly (V): 625 flex t4g.medium instances are on-demand while 1,911 are spot. U: why the pool, defined as on-demand, also runs spot.

• Ephemeral vs persistent (V/I): each label carries runs-on=<run_id>, and the runner names runs-on--i-<instance> (V) point to one machine per job (I). The only persistent capacity is the pool: weekdays 07:00-23:59 Central keep stopped: 20, hot: 6; off-hours keep stopped: 3. The config documents "stopped < 15 s, cold start < 25 s" (V).
• Startup latency (V, from 61 sampled runs): job created → started is p50 23-35 s across all events. p90 is 31-40 s on PR events but 264-315 s on merge_group, with a max of 1,308 s. Every wait over 120 s was clustered at about 20:05-20:19Z, on both pool=ci-2cpu-arm64 glue jobs and 4cpu guard-pins. That window held 80 merge_group and 78 PR runs. I: a capacity burst. It also falls shortly after CI_CAPACITY=west-only was flipped (19:57Z), but the cause is U.
• Containers / K8s (V): container: only in semgrep.yml:48; docker buildx+ECR only in deploy-renewal-worker.yml:364-392; docker run in rule9-recheck.yml:313. No kubectl/kubernetes/k8s/eks/helm in .github/infra (control: runs-on matched 69 files).
• Spot reclaim, new versus hosted (V): ci-followups.yml rerun-if-runner-killed, and ci.yml:3365 ("two spot reclaims … cost a full rerun and held deploys") moved push:main test-full to on-demand.

4.3 Caching and artifacts

• Backend is still GitHub's Actions cache, not S3 (V): actions/cache ×26, restore ×18, save ×8, setup-node cache: npm ×62; zero runs-on/cache or extras=s3-cache. ci.yml:123-131: s3-cache dropped in PR #8279 because its "ref/branch scoping was unverified". Each stack still has an S3 cache bucket with EnableCacheIsolation=false. I: buckets unused; isolation never answered.
• Budget (V, actions/cache/usage): 10.68 GB across 715 entries, against GitHub's 10 GB repo cap. I: LRU eviction is happening. Largest items: next-build on main 3.7 GB (2 entries, keyed by github.sha), node-cache (setup-node npm) on refs/pull/* 2.4 GB in 153 entries, node-modules+node-cache on main about 3.5 GB, and eslint-cache on main 520 MB in 538 entries (the key includes github.run_id, so each main run saves a new one).
• Key expressions (V): node-modules-${os}-${arch}-node24-${hash(package-lock)} ×43. It is saved only from refs/heads/main (ci.yml:855-858, "PR/merge-queue refs saving their own 294MB copies … blew the 10GB cache budget"). Also tsc-build-…-${sha}, next-build-…-${sha}, tsc-cache-…-${ref}, eslint-cache-…-${ref}[-${run_id}], pipeline-lab-llm-cache-…. I: GitHub scopes caches by branch, so PR and merge-queue runs read main's caches but write their own. The setup-node cache: npm saves are not covered by the main-only rule, which explains the 153 PR entries.
• cache-janitor.yml (V): daily 04:17; deletes queue-ref caches >3 h, PR caches idle >48 h, non-main duplicates of main keys, and older next-build/tsc-build on main (max 600).
• Artifacts (V): 40 upload-artifact@v4 and 5 download-artifact@v4 uses. These go to GitHub artifact storage. ci.yml has 3 uploads; nightly has 12.
• Persistence (I): no vitest/tsgo result cache; .tsbuildinfo and .next/cache ride GitHub cache by SHA. U: whether reused pool instances keep disk state.

4.4 What GitHub-hosted runners provided implicitly that now has to be supplied explicitly

• Image: pool uses ubuntu24-full-arm64 (V); other labels use a RunsOn default (U).
• Queueing and autoscaling: RunsOn plus EC2 quotas plus the pool schedule. push-pacer.sh throttles pushes at the source (ci-and-deploy.md:268 cites a "CI queue … 250 runs" episode that "pinned the RunsOn fleet") (V).
• Isolation: one instance per job (I); pool reuse is U.
• Cache: still GitHub's backend, now over its cap; S3 not adopted (V).
• No preemption: spot reclaim needs custom rerun tooling (V).
• Capacity: three stacks, a CI_CAPACITY switch, a run_id-digit split, and a 6-AZ rebuild not yet cut over (V).
• Network: public subnets, no NAT; checkout, cache and artifact traffic goes over the internet to GitHub (I).
• Visibility: org runner and variable APIs return 403 without admin:org (V).

§11 Queue economics: where work is thrown away

11.1 Volume, last 24 h (from 2026-10-01T22:00Z)

total_count figures are uncapped (V). The per-conclusion splits for PR, merge_group and review come from the API's newest-1000 cap, so they cover only the last ~10-18 h (V, caveat). Push is complete.

• V: 468 PRs merged in the window, which is about 4.8 merge_group runs per merged PR.
• I, extrapolated from n=15: 1,539 × ~56 exec-min ≈ 86,000 exec-min/day spent on merge_group runs that end cancelled. Fail-fast saves little: a cancelled run costs ~56 min against ~68 min for a passing one.
• I, cascade mechanism: the merge-queue ruleset is grouping_strategy: ALLGREEN, max_entries_to_build: 24, max_entries_to_merge: 16, check_response_timeout_minutes: 30, and the only required check is Nested Gate (V, rulesets API). Each queued entry builds on top of all entries ahead of it, so one red entry turns red every group built behind it, and those groups are rebuilt. The red jobs come in consecutive run-ID clusters (V: runs 37058672018-37058962771 all failed Full set, and 37068878224-37069655940 all failed Guard pins).
• Minutes (I): RunsOn bills instance-seconds × price; GitHub's rounded minutes apply only on the hosted fallback.

11.2 Duplicated test work

• The full suite runs twice inside each merge_group run (V): 6 Guard pins shards (4cpu, on-demand in the queue) plus full-set-one-machine (64 vCPU, step "Full guard pins, then full unit tests"). Both appear in the same job list (run 37068662351), alongside Unit Tests (affected).
• The same commit is tested on three SHAs (V). ci.yml triggers on push:[main], pull_request, merge_group, pull_request_review and workflow_dispatch (ci.yml:153-260). It is the only workflow with merge_group. Its workflow_call children (evals, semgrep, docs fence, pr-size-tier, sync-*) inherit those events. On push, merge-group-validated.sh skips work the queue already validated (ci.yml:506), so push runs are light (~9-37 exec-min, mostly the waiter jobs).
• test-full [1,2] did not run in any of the 61 sampled runs (V: absent). Comments say it runs on push:main mode=full and on review verdicts for hub-sized PRs (I).
• Shadow experiment (V): one-machine-shadow.yml runs on pull_request (1,055 runs/24h) and executes on 8cpu spot whenever the head SHA starts with 0-3 (about 25% of runs), with continue-on-error. U: its per-run minutes.
• post-merge-heavy.yml (V): cron 7,37 (45 runs/24h), with unit-full ×2 and guard-pins ×3 shards on main HEAD. This is another full-suite pass.

11.3 Installs, shards and global checks

• npm installs per ci.yml run (V): 10 npm ci sites, all behind a node_modules cache restore: checks, test, guard-pins ×6, test-full ×2, seed-cache, full-set-one-machine. So a merge_group run does up to about 9 restores, or 9 npm ci on a lockfile miss. deploy-lambdas.yml has 22 install sites (315 workflow_run runs/24h, V). Totals: nightly 16, harnesses 11.
• Matrix sizes (V): guard-pins 6, test-full 2, post-merge-heavy 2+3, plus 4-6-job trivial "name" fan-outs.
• Global versus affected (V/I): checks (type check, lint, DDB escapes, build) runs the full repo on every PR, merge_group and push. Its median was 2.7 min on PR and 12.4 min on merge_group. Unit tests are affected-only on PR (test → mode=affected/delta/full) and full in the queue.

11.4 Idle waiter jobs occupying instances (V, sampled durations)

Vercel Production Promote ~30 min (pool), Queue fail-fast ~15 min (1cpu), Preflight (debounce) ~3.3 min on push. ci.yml has 56-65 jobs per run, of which 44-52 are skipped. I: review runs (1,482/day × ~4 glue jobs) produce several thousand pool boots a day to deliver debounce and verdict results.

11.5 Concurrency inventory (V)

• ci.yml:331-333: group = workflow + event + action class + (run_id for push, otherwise ref). cancel-in-progress: ${{ event_name != 'push' }}. Push runs never cancel each other; superseded pushes cancel themselves in scripts/ci/main-run-superseded.sh (ci.yml:483).
• Job level: vercel-promote is in group vercel-promote-production with cancel-in-progress: false (ci.yml:4828-4830). I: GitHub allows only one pending job per group, so each newer push's promote replaces the pending one. That replacement is the source of the 80% "cancelled" push runs.
• Other workflows: 16 use cancel-in-progress: true (e.g. semgrep, one-machine-shadow, claude-review-recheck). deploy-lambdas, intent-evals and renewal-evals* cancel only on PR events. false on claude-code-review (1,117 PR runs/24h), auto-merge-all, deploy-renewal-worker, sync-* and post-merge-heavy. deploy-lambdas has 22 job-level groups and nightly has 12.

Caveats

• All per-run minutes come from samples: merge_group cancelled n=15, passing n=10, push cancelled n=10, push passing n=6, PR n=10, review n=10. Every 7th run was taken from the newest 1,000.
• The EC2 counts are a snapshot of roughly the last hour.
• The 530 runner registrations are evidence that runners are per-job, not a measure of fleet size.
• Who cancels a merge_group run, fail-fast or GitHub's queue invalidation, was not separated beyond the sample. Every sampled cancelled run carried its own red.

§5 Test topology — propflowai @ origin/main 864d9a57570

Tags: V verified, I inferred, U unknown. :N is a ci.yml line at the pinned SHA. Timings come from gh api …/jobs, 2026-10-02.

5.1 Inventory (V, git ls-tree origin/main)

5.2 What "affected" means

The selector (V). The selector is npx vitest list --filesOnly --changed $BASE (ci.yml ~2475, scripts/test-affected.sh, scripts/ci/merge-group-affected.sh). It picks two kinds of file:

• test files that changed, and
• test files whose static import graph reaches a changed file, using the merge-base diff.

Locally it also includes staged, unstaged and untracked files.

The base per event (V):

• PR and verdict-review: origin/<base.ref>, fetched fresh, with checkout fetch-depth: 0 (:2420). On pull_request, the checkout is GitHub's merge ref (I; guard-manifest-select.sh tests HEAD^2 for exactly this). So tests see base as of the push, not as of merge.
• merge_group: merge_group.base_sha.
• push:main: the previous main tip. If it is unusable, the run goes mode=full.
• Local (npm run test:affected): origin/main as last fetched. The script never fetches. It refuses an unresolvable ref or a missing merge-base.

The cap (V). AFFECTED_CAP=200. What happens over the cap depends on the event:

Zeros (V). classify-zero-selection.mjs splits two cases:

• Z1 (green): docs, config, or source outside coverage.include.
• Z2 (untested, merge block): changed source inside src/lib/, src/app/api/, agents/clara/lib/ or lambda/ (vitest.config.ts:487-493) that no test imports.

assert-selection-covers-changed-tests.sh fails if a changed test file was dropped. Selector crash → exit 1.

Gating (V, decide-test-lane.sh).

• Drafts skip the lane unless labelled run-tests.
• Dependabot PRs skip it and get test-full.
• Review events with no verdict skip it. So do verdicts on a SHA whose Unit Tests already concluded.

Blind spots:

1. Tests that read their subject with fs instead of importing it are invisible to --changed. PR #5471 caused a 14-hour deploy stall this way. CI compensates with:
   • (a) On PRs only, the "Always-run drift and decision tests" step runs every .drift.test. and decide-.test. file except 42 hard-coded slow ones (always-run-tests.mjs SLOW_EXCLUDED).
   • (b) Guard pins (5.3).
2. The always-run step's condition is mode ∈ {affected, delta, no-tests} (V). A hub-sized PR (capped) gets neither the affected run nor the drift set on its push. Only the fence subset of guard pins runs until a verdict arrives.
3. merge_group's affected lane has no always-run step (V, merge-group-affected.sh). It relies on the full-set job.
4. Local runs have none of this compensation. test-affected.sh is plain --changed. A local green says nothing about tests that read .sh, YAML, JSON or workflows.
5. Root config edits (package.json, vitest.config) probably match vitest's default forceRerunTriggers, which selects everything and escalates (I: no override in the config).
6. Never selected (vitest.config.ts excludes):
   • 15 EVAL_SUITES (opt-in RUN_EVALS=1)
   • the timing-quarantined mutation-control suite (nightly)
   • 3 ADVISORY_TESTS (advisory-tests.yml)
   • e2e/
7. Shallow history. The affected job aborts on a missing merge-base (:2465). The old --depth=1 empty-selection failure is closed on main (V, depth 0).

5.3 Guard pins (drift/runtime-config validation) — V

The list. guard-union-files.mjs builds one list, with floors checked per source, from:

• name filters (drift.test, drift-guard),
• derived fs-reader tests (readFileSync, git scans, tree walks), and
• reverse dependencies of the #alerts emitters.

It holds about 1,035 files, per a guard-select.mjs comment (I as of today).

What each event runs (guard-select.mjs):

• FULL on push, merge_group, review, dispatch and schedule.
• On a pull_request push, GUARD_PIN_KINDS=fence first drops about 330 bookkeeping pins. The PR then runs a subset:
  • always-on guards matched by name (guard|registry|inventory|fence|pins|manifest),
  • always-on guards whose content walks directories or reads workflows or registries, and
  • guards whose source text names a changed path, directory or basename.

A PR push never goes FULL, even past 300 changed files.

Shards and timing. 3 shards on PRs and 6 on merge_group (observed). There is no --passWithNoTests, and each shard checks the list's count and sha256. Each shard takes about 2.1–3.0 min on PRs and 2.6–3.6 min on merge_group.

The integration gap it documents. On 2026-10-01 there were 122 merge_group Guard-pin reds. Each was on a bookkeeping pin whose fact the batch's PR had moved, under a green PR run. The merge-queue full set on fresh main is the only lane that sees a pin moved by a sibling merge.

5.4 Category table

5.5 Result reuse (V)

• By tree, across commits. merge-group-validated.sh runs in push:main preflight. It matches the pushed commit's tree SHA against the last 20 successful merge_group runs. On a match it skips test, guard-pins and test-full (:2327, 3130, 3326).
  • Example: push run 37063383054 ran only CI Checks (1.5 min) before promoting.
  • This is the only reuse that crosses commits.
• By SHA only. Verdict runs skip if Unit Tests already concluded on the SHA. Guard pins skip on a prior green full set. Any new push, including a merge of main, re-runs everything.
• Caches (node_modules by lockfile hash, tsc, eslint, next) save time; they are not reused results.

5.6 Which lanes catch integration failures

Queue health (V).

• Last 100 merge_group runs: 44 success, 39 cancelled, 17 in progress.
• Last 100 push runs: 21 success, 77 cancelled (superseded by design).
• Queue fail-fast (14.7–15.3 min) cancels a group the moment any verdict job goes red.

5.7 Unknowns

Percentiles (small samples), live guard-union size, real eval durations, test-full cadence, and whether the full set's quarantine or known-red exclusions mask real breaks.

PropFlow CI/CD forensic map: §8 Deploy boundaries, §9 Freshness/staleness

Workflow and script files were read at 864d9a57570. The PR-lifecycle lookup ran after a mid-task fetch, against origin/main = 89ea1dc5e2. Live API and AWS reads (read-only) were taken around 2026-10-02 21:50Z. Tags: V = verified (file:line or command output) · I = inferred · U = unknown.

§8 Build & deployment boundaries

8.0 Headline

• Production is one coupled, batched release train with independent per-unit gates. All units hang off the same push:main CI run. The web promote is a job inside ci.yml. Lambdas and the worker trigger on workflow_run: CI completed, gated on conclusion==success && event==push. V (deploy-lambdas.yml:86-88,122-124; deploy-renewal-worker.yml:35-36,107-109; ci.yml:4762)
• The code says the coupling is deliberate: "a failed WEB build now blocks the Lambda lane outright… a merge is one unit". V (ci.yml vercel-promote comment)
• Exception: the voice (ElevenLabs) and DLQ-alarm syncs run on a raw push: main path filter. They are not gated on CI and not batched (8.4). V

8.1 Next.js web app (Vercel)

src/ → timeout 1200 npm run build (vercel.json:2) → Vercel deployment → propflowai.co alias → Vercel functions + ~30 Vercel crons. V

• Vercel does not build main through git integration. scripts/vercel-ignore.sh skips main, and skips previews unless the commit says [preview]. The skip lives in the dashboard Ignored-Build-Step setting. vercel.json has no ignoreCommand, and that is deliberate: one there would cancel the API promote. V
• Prod is a held, batched promote created through the Vercel API, job vercel-promote (ci.yml:4762):
  • It runs only on push, after checks (type check + build) and test-aggregate are green (4771-4776). V
  • Concurrency group vercel-promote-production, cancel-in-progress:false (4829-4830). One deploy runs at a time, one waits, and a third arrival cancels the waiting one. V
  • It waits for a 30-min batch window (scripts/ci/promote-window.env: PROMOTE_WINDOW_DEFAULT_MINUTES=30; the repo var PROMOTE_WINDOW_MINUTES overrides it). The job allows up to 40 min of waiting (timeout-minutes: 80). V
  • It deploys the newest green tip, not necessarily its own SHA (decide-promote-target.sh), checks ordering against the SHA production is actually serving, then warms the site and requeues the tip if it was cancelled. V
• promote-now.yml: a manual dispatch whose run tells a waiting promote to skip the window. V
• main-preview.yml: on a green main CI run, creates an API preview deployment and aliases propflow-main.vercel.app / main.propflowai.co (never backwards). V
• prod-freshness-alarm.yml: a 30-min cron that compares main's green tip with the SHA Vercel production is serving and pages on a stall. V
• deploy-queue-watchdog.yml: a 10-min cron that finds deploy runs starved of runners. It auto-cancels only renewal-worker runs. V

8.2 Lambdas (deploy-lambdas.yml, 4,280 lines)

lambda/<name>/ + the src/lib/ it imports → npx tsx lambda/<name>/build.ts (esbuild) → dist/lambda.zip → aws lambda update-function-code → publish-version → update-alias live. This is raw AWS CLI: no SAM, CDK or CloudFormation. V**

• 16 lanes are in the gate registry (LAMBDAS=, about line 230). lambda/vercel-log-drain exists in the tree but has no lane here; its deploy path is U. V
• The gate: one compute-deploy job runs scripts/lambda-deploy-gate.ts per lane. It is an esbuild import-graph check over BEFORE..AFTER, where BEFORE is the head of the previous successful push:main CI run. Cancelled or red runs therefore fold into the next range. If the gate can't decide, the lane deploys anyway (fail-open). V (113-260)
• Each lane has its own job-level concurrency group (e.g. lambda-update-propflow-inbound-processor-prod). sns-slack-forwarder deploys as a canary inside ci.yml (5075). V
• In practice nearly every lane deploys: 13–15 of the 15 non-canary lanes in 18 of the last 22 successful runs. Run 37061506704 deployed all 15. V A per-lane [bundle-skip] can turn some of those successes into no-ops; how many is U.
• A lambda run takes 6–8 min. V

8.3 Renewal / Temporal worker (deploy-renewal-worker.yml)

src, agents, services/renewal-worker, infra/temporal-worker, packages, config, lockfiles and a few others → source hash (git rev-parse HEAD:<paths> | sha256sum, 202-217) → arm64 Docker image → ECR propflow-renewal-worker:<sha12> → ECS Fargate service renewal-worker on cluster propflow-renewal-worker-prod (new task-def revision, ecs wait services-stable). After success the image gets the tag deployed-<env>-<srchash>. V

• HEAD:src is in the hash, so almost every code merge rebuilds the worker. I
• The target is the newest green commit, bounded between the trigger SHA and the main tip (129-160). V
• Successful runs take 13–34 min. V
• Live check: task-def :4393 runs image :399a99e259e1, the same SHA as Vercel prod (deployment 6818693735, 21:28Z). V The filtered runs list showed nothing for this workflow after 19:45Z although a deploy clearly happened, so run listings are unreliable as a freshness signal. V

8.4 Third-party config written on push:main (not CI-gated, not batched)

• The apply jobs run when github.event_name == 'push'. The ci.yml workflow_call copies (6066-6094) need plan, and plan runs only on pull_request/merge_group (ci.yml:5987). So on push, the standalone trigger is the only apply path and nothing applies twice. V
• The result: voice config goes live minutes after merge, before that commit's CI finishes and outside the web batch. The voice agents and the web routes they call (e.g. /api/voice/personalization) can briefly run different commits. I

8.5 Other units and coupling

• services/ contains only renewal-worker. infra/runs-on-flex (CI runners) and infra/temporal-worker (the task-def template, part of the worker hash) have no IaC apply workflow; how they are deployed is U.
• post-merge-heavy.yml runs the full suite after merge and is not a deploy gate. V
• What couples the units:
  • One trigger: a single push-CI verdict gates the web app, the Lambdas and the worker. V
  • Shared code: Lambdas bundle src/lib, and the worker image copies all of src/. V
  • One prod table: a single DynamoDB table, propflow-prod. Version skew between units is bounded by the batch cadence. I
  • Shared env/secrets: U.

8.6 Measured deploy cycle (24h to 21:50Z, 2026-10-02)

• Throughput: 466 first-parent commits on main, 465 of them (#N) squash PRs. gh pr list merged:>= returns 469. That is about 4.5× the "~100/day" premise. V Per-day counts this week: 212, 153, 381, 346, 430, 523, 386. V
• Push CI runs: 300. Conclusions: 238 cancelled, 46 success, 12 failure. "Cancelled" means a superseded promote job; the tests were green (run 37068378536: every job succeeded except the promote). V There are fewer runs than commits because the queue merges up to 16 entries per push and paths-ignore skips docs-only commits. V/I
• Successful push-CI wall time: median 39.6 min, p90 57.8 (includes the batch wait). V
• Vercel prod: 39 deploys in about 18.5h. Median gap 35.0 min (max 86). Median 5 commits per deploy (max 42). V
• Landed on main → live on prod (web): median 53 min, p10 24, p90 90, max 109 (n=197 push-tip commits). V
• Lambdas finish 6–8 min after CI completes. The worker finishes 13–34 min after. V
• End-to-end, enqueue → live: about 35 min in the queue + 53 min ≈ 1.5h median. I

§9 Freshness / staleness

9.1 Branch creation

• ~/.claude/hooks/guard-bash.sh:669-682 denies git worktree add -b unless the command names origin/main. Lines 722ff deny depth-limited fetches. V
• scripts/worktree-setup.sh:75-94 fetches and rebases the main checkout, then runs worktree add -b with no explicit base (94). .husky/post-checkout:428-432 then fetches and fast-forwards a fresh branch to origin/main. V MEMORY says the script is broken on the mini, so agents use a bare worktree add. I

9.2 While the PR is open, nothing refreshes it

• The only PUT …/update-branch is a capped self-heal for a wedged Unit Tests suite (auto-merge-all.yml:3467). V
• stacked-base-sweep.yml is a nightly job that only reports. stacked-pr-guard.yml only disarms auto-merge on stacked PRs (39). V
• Doctrine (CLAUDE.md): do not merge main into a PR unless it is CONFLICTING or needs a fix that is on main. V In 40 sampled merged PRs, 0 contain a merge commit. V
• Exception (auto-merge-all rule 9): a PR that changes a check is held until the base is brought in. V (~907)
• .husky/pre-push (158-180) reports the behind-count as advisory only and never fetches, so the count can be stale. V The push-pacer (pre-push:199 → scripts/ci/push-pacer.sh) rate-limits bursts and fails open. V

9.3 Which main a PR is tested against

• PR-event jobs check out with no ref (24 of 25 checkouts; the only one with a ref is ref: main at 7044). They test refs/pull/N/merge, which is the PR merged into main as of that event. Confirmed in ci.yml:2507-2510. V
• Branch protection: strict: false, so a PR does not have to be up to date. Required checks: Build, Type Check, Unit Tests, Regression Gate — Leasing, Lint, review. V
• Ruleset "Require CI": requires Nested Gate, plus a merge queue: SQUASH, build up to 24 entries, merge up to 16, min-wait 2 min, ALLGREEN, 30-min check timeout. V
• The queue is the first test against current main plus the PRs ahead of it. It runs affected tests against merge_group.base_sha and Full set (one machine) (6565). A hub-sized selection "executes nothing here… full suite runs after merge" (merge-group-affected.sh). V
• In the queue, review and the Leasing gate are pass-by-design (merge-group-by-design), so the LLM review never sees the combined tree. Live ElevenLabs drift tests are skipped in the queue. V

9.4 Survival across a new SHA

• Checks: per-SHA. The gates read pull_request.head.sha (e.g. 2511), and any push requires a fresh run. V
• Review: since 2026-09-16 a clean verdict does not carry forward. claude-review-recheck.yml (on synchronize) carries only a BLOCKING verdict, as a separate Carried-forward BLOCKING verdict check-run (118). supersede-carried-review.yml replaces that stamp once a real verdict lands. Every rebase or merge of main therefore costs a full LLM review round. V

9.5 Measured lifecycle (40 latest merged PRs, #15564–#15754)

V (gh pr list + pulls/N/commits + git rev-list --first-parent). "Base" is the parent after any rebase, so this undercounts the original branch age. I

• Main commit rate: about 19.4/hr averaged over 24h. Hourly buckets range from 1 to 36. V
• About 28 commits (p90 58) land on main between a PR's base and its merge. How many land after its last PR-level CI event (its last refs/pull/N/merge snapshot) is U. Only the queue tests the PR against them. V/I

9.6 Where staleness enters

1. Branch creation: fresh, enforced by the hook plus the post-checkout fast-forward. V
2. Open PR (~2.5h median): tested against main as of its last event and never refreshed. V
3. Queue: current main, but affected tests only and no re-review. V
4. After merge:
   • Web prod lags main by a median of 53 min. V
   • Lambdas and the worker trail the same green CI run by 6–34 min. V
   • Voice config goes live ahead of CI. V/I
5. Superseded promotes: about 80% of push runs end "cancelled". Lambdas and the worker fire only on the ~46 green runs a day; diff-range folding covers the commits in between. V

§10 Failure forensics: PropFlow CI/CD

Collected 2026-10-02 ~22:00 UTC, read-only; main tip 429eca86187. Tags: [V] verified from logs, annotations or the API; [I] inferred; [U] unknown. Failures were classified from failing-job annotations and log greps, not from run conclusions. Excluded: claude-code-review verdicts, and cancellations from cancel-in-progress (push:main had 2,063 of these since 09-18 [V]).

0. Three facts that change how every number reads

1. Since 10-01 20:03 UTC, queue failures are recorded as cancelled. [V] PR #14934 added a Queue fail-fast job that cancels the whole run once any verdict job is red. Merge-queue runs ending failure went from 174 on 10-01 to 0 on 10-02, while cancelled went to 1,377. In a sample of 180 cancelled runs from 10-02, 170 (94%) contain a real failed job [V].
2. The queue was not on the whole time. [V/I, from PR timelines] It ran 10-01 from 11:00 to 20:00, was off from 20:00 to 23:00 (84 PRs merged directly in that hour), and has been on since 10-01 23:00. Below, "pre-queue" means before 10-01 11:00 UTC.
3. Run counts are not incident counts. [V] One broken test produces many red runs, because:
   • every merge group queued behind a bad entry fails too;
   • every push to main re-fails until it is fixed;
   • "label-only" PR runs re-state a failure that already exists.

1. Merge queue (all post-queue)

Runs per day [V]:

Reruns that came back green [V]:

• 10-01: 17 of 369 successes were attempt 2 or later (4.6%).
• 10-02: 0 of 500 sampled.

1a. Before fail-fast (178 failed runs, all classified):

Incidents: these 178 runs collapse to about 12 distinct incidents.

• 2 were inherited from main (the DDB pair).
• About 9 are clusters of one test failing across many PRs. This data cannot tell whether a cluster came from a bad entry at the head of the queue (cascade) or from a stale or broken base [U].
• 1 is a single PR's own bug.

1b. After fail-fast: sample of 171 cancelled runs that contain a failed job. The sample is skewed: 100 come from 19:00–22:00.

CI config churn is a cause in its own right [V]: about 20 merged changes to ci.yml between 09-30 and 10-02, including two reverts (#15220, #15372). Several of them change which tests the queue runs. [I] Some clusters are tests that never ran before merge until the queue started running them: a coverage gap showing up, not two PRs conflicting.

2. Red main (push:main ci.yml, 09-18 to 10-02)

703 failed runs [V]: 671 on attempt 1, 32 on attempts 2–3. Of about 4,100 total runs. Worst days: 09-29 (107/338) and 10-01 (224/509). 10-02 had 6/246. Reruns that came back green: 14 of 500 sampled successes (2.8%) [V].

By period [V]:

Caveat [V/I]: PR #14455 (09-30) and PR #15194 (10-02) made push:main skip test lanes the queue already ran. So the drop is partly real prevention and partly main testing less. This data cannot separate the two [U].

163 incidents (same test, gaps under 6 hours). Incidents lasting more than one run were red for a median of 0.5 h, p90 4.2 h [V].

\*A CI self-test emits about 70 deliberate ::error lines, which crowds the real failures out of the annotations. A 50-log sample of this bucket resolves mostly to user-route-inventory, entity-conversation-routes and active-thread-invariant.

Was main red on code that was green on its PR? For the 22 largest test incidents I looked at the PR behind the first failing run [V]:

• 18 of 22 had Unit Tests=success at the PR head.
• Guard pins was usually skipped on the PR.
• 3 had no Unit Tests result.
• #13915 shows Unit Tests=failure only, i.e. it merged red.

Two caveats:

• Cancel-in-progress can hide the true first-red commit [I].
• "Green on PR, red on main" mixes two different things: (a) a real overlap between PRs, and (b) the PR lane never ran the test (affected-only selection, Guard pins skipped). The skipped-guard data suggests (b) is common [I].

73 "red main" or "collision" fix commits on main since 09-18, read by their titles [V titles; I categorisation]:

3. PR runs (pull_request ci.yml)

Failed / total by day [V]: 09-30 405/1,724; 10-01 460/1,589; 10-02 208/1,398. Sample of 30 failed runs, every 10th, from 10-01 and 10-02:

†Stale-base candidates: #15648 failed the customer-fence baseline while touching 0 related files [I]. #15094, itself a red-main fix PR, failed on the inventory [I].

So about 55% are the PR's own fault, and mostly policy violations rather than functional bugs. About 13% are prone to a stale base, and about 23% are noise. The sample contains no clear flake [V, n=30].

4. Queue removals vs merges since 10-01

Source: GraphQL RemovedFromMergeQueueEvent.reason. Note that the REST removed_from_merge_queue event also fires on successful merges. Coverage is 1,000 of 1,263 PRs, so counts are understated by up to about 20%.

• 719 PRs entered the queue (990 entries). 534 merged; 425 (59%) on their first entry.
• Non-merge removals [V]:

• 94 of the manual removals came from the gera/fede accounts between 19:00 and 21:00 on 10-02. This looks like a bulk dequeue [I]; who did it is [U].
• Queue exits excluding manual (n=676): 79% merged, 12% failed checks, 9% textual conflict.
• About 1,460 failing merge-group runs produced only 80 failed_checks removals. Each bad entry fails the speculative groups behind it, which are then rebuilt rather than ejected [I].

5. Against the hypothesis ("~50% stale-main/overlap, the rest build/runtime/config, flakes rare")

• Flakes are rare: holds. 4 of 73 red-main fixes are flakes; 0 of 30 in the PR sample. Reruns that came back green are 0–4.6% of successes, and that figure includes infra retries. Quarantine (#15203) may be absorbing some flakes [U].
• Classic build/runtime/config failures are small.
  • Compile: about 0–1%.
  • Deploy: about 2%.
  • Infra: about 3–7% (vCPU quota, spot, disk, CI rate limits).
  • The exception is churn in CI itself.
• Integration-type failures exceed 50% on main and in the queue, but they are not mostly textual or two-PR overlap. Textual conflicts are 9% of queue exits, and only 7 of 73 fix commits name two colliding PRs. The dominant mechanisms are:
  1. A PR merges green because its lane never ran the test (affected-only selection, Guard pins skipped, a scoped DDB-escape gate). Everyone after it inherits the red. This is verified for #14896 and inferred for #14850.
  2. Policy tests pinned to exact counts, line numbers, manifests or baselines. These break whenever two PRs touch the same inventory.

On PRs, the PR's own policy violations dominate.

6. Missing data

• Per-run classification of the ~1,290 failures hidden by fail-fast on 10-02: I only have a skewed 171-run sample.
• Culprits for most clusters. Only #14896 is verified, and #14850's cluster end is unexplained.
• Whether each failing test actually ran in the culprit PR's lane. This would separate a coverage gap from a real overlap; it needs per-PR affected-test lists.
• 29% of main failure runs are unclassified (annotation flood).
• Which check failed in each failed_checks removal, and which files conflicted in each merge_conflict removal. Neither is exposed after the fact.
• PR coverage is capped at 1,000.