# CI waste audit — PropFlow-Technologies/propflowai, 2026-08-31 → 2026-09-30

Read-only. Raw data cached in `scratchpad/audit30/` (runs.tsv, h2.tsv, prs_*.json, pr/*.json, ce.json).
Labels: **FACT** = counted from API data; **EST** = derived/extrapolated.

## 0. Data coverage and caveats (read first)
- The runs API returns at most 1,000 runs per query. Listing in 3-hour windows gave **210,500 runs**, but 184 of 248 windows were capped, so this is a *sample* (newest 1,000 per 3 h), not the population. Shares below are from the sample.
- True volume (FACT, `total_count` per hour, split to 10-min when capped): **471,652 runs from Aug 31 to Sep 22** (543 of 720 hours counted). Peak day Sep 18: 44,128 runs. Sep 23–30 not counted (rate-limited). **EST month total ≈ 700–750k runs.**
- My listing pass burned the shared 5,000/h core API budget and then hit GitHub's secondary limit (~15:33 UTC). Consequences: per-job data (machine minutes, 8-core vs 2-core per job) could NOT be sampled (3 runs only; unusable); GitHub billing endpoint was refused (rate limit, not permission — retry later). Machine cost is therefore taken from AWS Cost Explorer, and "minutes" below are **run wall-minutes** (updated_at − run_started_at), not machine-minutes. For CI (dozens of parallel jobs) machine-minutes are several times wall-minutes.
- `gh pr list` capped at 1,000 per 10-day window: 2,909 merged PRs retrieved (fede 1,273, gera 1,636); days Sep 11–15 and 21–26 are missing. EST true merged ≈ 4,000–4,500.

## 1. Lifecycle per PR (80 merged PRs, 40 fede + 40 gera, random, merged Sep 2–28)
Method: GraphQL timeline per PR; a "verdict" = each non-dismissed PR review by `claude` (290 verdicts); green = APPROVED or 🟢/LGTM; deploy = end of the first successful `CI` push run on `main` (holds the `vercel-promote` job) created after the merge (EST: sampled runs, coverage gaps bias it upward).

| Segment | all median / p90 (h) | fede | gera |
|---|---|---|---|
| Open → first bot verdict | 0.10 / 0.27 | 0.09 / 0.15 | 0.11 / 0.72 |
| First verdict → last 🟢 | 0.31 / 1.58 | 0.48 / 1.33 | 0.11 / 5.48 |
| 🟢 → merged | 0.07 / 0.31 | 0.07 / 0.15 | 0.09 / 0.96 |
| **Open → merged** | **0.65 / 4.01** | 0.58 / 2.49 | 0.73 / 10.88 |
| **Merged → in production (EST)** | **1.13 / 2.46** | 1.13 / 2.38 | 1.20 / 3.10 |
| Review verdicts per PR | 2 / 10 | 3 / 11 | 1 / 8 |
| Pushes per PR | 2 / 7 | 2 / 6 | 2 / 9 |
| PR size (lines) | 383 / 1,586 | 442 / 1,442 | 344 / 1,755 |

Where the time goes: for the median PR, **deploy (~1.1 h) takes longer than the whole open→merge (0.65 h)**. The 30-min promote batching plus a full push-CI run on main (avg 14 wall-min, 21% cancelled/failed) stacks up. In the tail, review rounds dominate (p90 first-verdict→green 1.6 h; 10 rounds). Waiting for auto-merge after 🟢 is small (4 min median). Runner queueing could not be measured (no job data).

## 2. Runs by outcome (sample of 210,500 runs)
| Conclusion | Runs | Share | Wall-min |
|---|---|---|---|
| success | 112,473 | 53% | 180,142 |
| skipped (zero work) | 80,861 | **38%** | 4,898 |
| cancelled | 12,780 | 6% | **47,317** |
| failure | 4,159 | 2% | 35,149 |
| startup_failure | 20 | — | 72 |
| queued/in-progress at fetch | 207 | — | — |

- **Cancelled wall-minutes = 17% of all wall-minutes** (47k of 272k). Biggest sinks: CI 24,171; Auto-merge 4,283; Quality Gate POC 2,977; Review re-pin 2,583; Claude Code Review 1,935; Branch Preview 1,621; voice-comprehensive 1,596; Self-review 1,392; Deploy Renewal Worker 1,147; Clara Behavior Evals 1,105 (more cancelled than successful minutes).
- Zero-work runs: 80,861 skipped (38%) — mostly event fan-out that evaluates an `if:` and exits. Top: Claude Code 23,750 (97.5% skipped), Rerun on runner shutdown 10,211 (89%), Auto-merge 9,981, Docs-only fence 7,822 (97%), Claude Review Check Override 6,039 (74%), Dependabot Review Gate 4,785 (99%), Bot PR Required Checks Gate 3,684 (100%), Review re-pin 3,662, CI 2,229, Rerun review on infra failure 2,086 (73%), WO Creation E2E 638 (99%).
- Reruns (run_attempt > 1): 529 in sample (CI 294, Claude Code Review 110, Self-review 48).

## 3. Workflows with ≥ 50 runs in the sample (trigger, wall-min, does it change a merge?)
| Workflow | Runs | Wall-min | Trigger | Verdict |
|---|---|---|---|---|
| Auto-merge on green (all PRs) | 45,921 | 23,921 | workflow_run 21.5k, review 9k, PR 7.9k, dispatch 7.6k | Needed function, **rethink**: fires on every other workflow's completion; 22% of all runs |
| Claude Code (@claude) | 24,359 | 2,565 | issue_comment/review/review_comment | 97.5% skipped; **narrow trigger** to comments containing `@claude` |
| CI | 12,587 | 103,117 | PR 5.4k, **review 5.6k**, push 1.5k | Required. **Delete the `pull_request_review` trigger** (45% of CI runs; 2,741 full successful reruns + 432 failures, 27,498 wall-min) |
| Review execution re-pin | 11,998 | 6,955 | workflow_run 7.9k, review 4.1k | Review machinery, **merge into review workflow** |
| Rerun on runner shutdown | 11,527 | 1,054 | every workflow_run | 89% skipped; runs on GitHub-hosted `ubuntu-latest` (conflicts with no-hosted-runners rule) |
| Claude Review Check Override | 8,165 | 1,204 | review | 74% skipped; **merge** |
| Docs-only fence | 8,042 | 436 | PR (reusable) | 97% skipped; **delete or inline as path filter** |
| Branch Preview Domain | 6,407 | 4,921 | every PR push | Not a merge gate; **run on `opened` only** |
| Claude Code Review | 5,049 | 32,650 | PR 4.9k, dispatch 128 | Required. 20% failure, 7.5% cancelled — **rethink** (see §7) |
| Dependabot Review Gate | 4,820 | 258 | pull_request_target | 99% skipped; **filter to dependabot actor at trigger** |
| Nested Package CI | 4,772 | 3,290 | PR | **path-filter** |
| Quality Gate POC | 4,687 | 7,339 | PR | A "POC" running on every push; 41% of its minutes cancelled — **delete or nightly** |
| Semgrep SAST | 4,435 | 7,799 | PR | Only x64 job; **nightly + on security paths** |
| PR body gate / Self-review / PR size tier / PR file collisions / CI Dispatch Guard | 4,239 / 4,100 / 2,357 / 2,418 / 4,051 | 2.8k / 8.0k / 2.0k / 0.8k / 1.9k | PR | Five separate cheap gates, each a machine boot; **merge into one job** (Self-review already runs body+size) |
| Bot PR Required Checks Gate | 3,686 | 197 | workflow_run | 100% skipped; **delete** |
| Lambda bundle check | 3,477 | 4,766 | PR | **path-filter** to lambda dirs |
| Turnover Intake Regression Harness | 3,249 | 6,280 | PR | **path-filter** |
| Claude Review Recheck | 3,182 | 1,184 | PR | review machinery, **merge** |
| Rerun review on infra failure | 2,862 | 440 | workflow_run | ubuntu-latest; 73% skipped; **merge** |
| Deploy Lambdas | 1,803 | 4,209 | workflow_run 1.6k | path-gate at trigger |
| Clara Behavior Evals | 1,771 | 1,745 | PR | cancelled-min (1,105) > success-min (571) |
| Deploy Renewal Worker (Fargate) | 1,686 | 10,833 | every main CI completion | 400 cancelled, 445 skipped; **trigger only when worker paths change** |
| Announce deploys to Slack | 1,684 | 328 | workflow_run | 38% skipped; fold into promote job |
| Review Verdict Watchdog | 1,316 | 1,379 | schedule (~every 30 min) | exists because review machinery is fragile |
| Production Freshness Alarm | 1,284 | 384 | schedule | keep |
| Quality Replay Typecheck, Portfolio harness, CI (docs), AppFolio loop harness, Pipeline Lab Evals, Claude Type Sync, Sync ElevenLabs specialists, voice-comprehensive (590 PR runs, 7,291 min, 196 cancelled), Require eval diff, One call one email, identity harnesses | 90–1,218 each | 145–7,291 | mostly PR | All per-push; most belong behind path filters. Portfolio harness fails 21% of runs |

"Copied, not rethought" evidence: 15+ small single-purpose workflows exist only to react to other workflows (re-pin, override, recheck, rerun-on-infra, rerun-on-shutdown, watchdog, bot-PR gate, dependabot gate, docs fence, auto-merge fan-out). Together they are **~60% of all runs**.

## 4. Per-push cost model (EST)
AWS Cost Explorer, us-east-1, EC2 by USAGE_TYPE, Credit/Refund excluded. Sep 28 and 30 are partial (CE lag / today); **Sep 29 is the only full RunsOn day**:
| Sep 29 | $ |
|---|---|
| EC2 compute + other, total | 266 |
| 8-core (`*.2xlarge`) | 150 — **on-demand 97 (c7g.2xlarge, 334 h)**, spot 53 |
| 2-core (`*.large`) | 69 — **on-demand 44 (m6g/m6a.large, 561 h)**, spot 25 |
| t4g.medium + CPU credits (likely RunsOn control plane / other infra, not jobs) | 38 |
| On-demand share of compute | **59%** ($157 of $266) |
- Run-rate: CI runners ≈ $219/day → **≈ $6.6k/month**.
- Per merged PR (Sep 29: 341 merged): **≈ $0.64**. Per push (EST 3 pushes/PR incl. main): **≈ $0.20**.
- Boot overhead: per-job data unavailable. With ~25–35 machines per full CI push (ci.yml has 55 jobs) and an assumed 90 s boot, boot is ~40 machine-min per push; at an 8–10 min full run this is **EST 15–25% of CI machine time**.
- 334 on-demand 8-core hours in one day means spot capacity is not being obtained for most 8-core jobs (price-capacity-optimized falls back to on-demand).

## 5. GitHub side
- Actions cache: 396 caches, **10.72 GB** — above GitHub's 10 GB per-repo limit, so caches are being evicted continuously (thrash; cache hits unreliable).
- Billing usage (`/orgs/.../settings/billing/usage`): **not retrieved** — refused with a rate-limit error after my listing pass; retry after the limit resets. Before/after dollars therefore not comparable yet. `rerun-on-runner-shutdown` and `rerun-review-on-infra-failure` still run on GitHub-hosted `ubuntu-latest` (≈14k runs in the sample) and `voice-replay` on `ubuntu-24.04-arm`.
- Side-cost: the event fan-out itself consumes the shared GitHub API budget (every orchestration run calls the API); this audit hit the limit in ~10 min.

## 6. Top 10 waste items (ranked by EST $/month; $ uses the $6.6k/month runner run-rate)
| # | Item | Evidence | Fix | $/mo EST | Risk | Size | Kind |
|---|---|---|---|---|---|---|---|
| 1 | On-demand fallback on 8-core and 2-core | Sep 29: $141 on-demand vs $78 spot; 334 OD c7g.2xlarge h | Widen spot families/AZs, allow 8cpu→spot-only with retry, cap OD | 1,500–2,000 | Evictions (ci.yml notes 4cpu spot was 100% evicted) | M | rethink |
| 2 | CI re-runs on review approval | 5,634 CI runs on `pull_request_review` (45% of CI runs), 27.5k wall-min, same SHA already tested | Remove the trigger; approval gates via required-check status, not a rerun | 700–900 | Low: the SHA is unchanged | S | delete |
| 3 | Cancelled CI | 24,171 of 103,117 CI wall-min (23%) cancelled | Run cheap gates first, start shards only after them; debounce pushes within ~60 s | 500–700 | Low | M | rethink |
| 4 | Per-push harnesses without path filters | Semgrep 7.8k, Quality Gate **POC** 7.3k, voice-comprehensive 7.3k, Turnover 6.3k, Lambda bundle 4.8k, Portfolio 3.8k wall-min | Path filters; POC + Semgrep → nightly | 400–600 | Missed coverage if filters wrong | S | delete/move |
| 5 | Review bot failures and supersessions | CCR 1,004 failures (20%), 378 cancelled; 10.1k of 32.6k wall-min wasted; plus model spend | Fix infra-fail cause, cancel on new push *before* the model call, stop re-fires | 300–500 + model $ | Low | M | rethink |
| 6 | Deploy Renewal Worker on every main CI | 1,686 runs, 10.8k wall-min, 400 cancelled | Trigger only when worker paths change | 200–300 | Low | S | rethink |
| 7 | Orchestration fan-out (auto-merge, re-pin, override, recheck, rerun-on-infra, rerun-on-shutdown, watchdog) | ~85k runs in sample (~40% of all), 36k wall-min, 2-core boots each; API budget exhaustion | Replace with one merge/review orchestrator (GitHub merge queue + one review workflow) | 200–400 (+API headroom) | Medium: touches merge guard (dangerous-diff floor) | L | rethink |
| 8 | Five separate PR metadata gates | PR body, Self-review, size tier, file collisions, dispatch guard: 17k runs, 15.5k wall-min, 1,642 Self-review cancels | One "PR gates" job | 150–250 | Low | S | merge |
| 9 | Always-skipped workflows | Docs-only fence 97%, Dependabot gate 99%, Bot PR gate 100%, Claude Code 97.5%, WO E2E 99% skipped: ~40k runs | Delete or move the condition into the `on:` filter | <100 (noise, API, UI clutter) | None | S | delete |
| 10 | Deploy latency / promote batching | Merge→prod median 1.1 h vs open→merge 0.65 h | Promote from the merge-queue SHA directly; skip duplicate push-CI on main when merge-queue already tested it | time, not $ (≈ 1 h per change) | Medium | M | rethink |
Also: Actions cache over 10 GB (thrash) → move to RunsOn S3 cache (S, time not $); Branch Preview Domain on every push (6.4k runs) → `opened` only (S).

## 7. Code-review waste
Method: 80-PR GraphQL sample (290 review records) + 5,049 Claude Code Review runs in the run sample.
- Verdicts per merged PR: **median 2, p90 10** (fede 3/11, gera 1/8). Review workflow runs per PR in the run sample: median 1, p90 3 (sample under-counts).
- Superseded/wasted runs: 378 cancelled (7.5%) + 1,004 failed (20%) of 5,049 = **27% of review runs produced no verdict**; 10.1k of 32.6k wall-min.
- Skip-passed (docs/workflow-only etc.): 334 runs (6.6%) skipped.
- Re-fired: 128 manual `workflow_dispatch` runs (2.5%); machinery around it in the sample: Recheck 2,935 successful runs, Check Override 2,112, Re-pin 7,640, Rerun-on-infra-failure 775, Watchdog 1,315.
- Runner: `claude-code-review.yml` runs on the 2-core RunsOn shape (not 8-core); review cost is mainly model spend.
- Reviewer model spend (219 footers parsed): tiers medium 140, large 30, tiny 20, huge 9, mega 5; **avg turns: tiny 10.5/30, medium 15.7/60–80, large 26/100, huge ~29, mega ~35**; **0 turn-cap hits**. Budgets are 3–5x what is used. Model: claude-opus-5 then claude-opus-5-5. Dollar spend not in logs.
- 🔴 (CHANGES_REQUESTED/blocking) verdicts: 30; **29 followed by a push**. "Next review says still open": 0 in review records; 14 of 68 by text search of summary comments (weak: summary comments are edited in place, so older text is lost).
- 🟡-only verdicts: 28; **20 (71%) were followed by a push before the next verdict** — against the 09-29 rule.
- Summary comments are sticky and edited in place, so per-round timestamps exist only on review records (comment `createdAt` shows the first round only).