Proposed — pending Fede's review
AI code review: the evidence
Measurement follow-up to the research brief · 2026-08-09 · every number below traces to review-harness/analyze/REPORT.md, data/analysis.json or data/actions-cost.json. Numbers not yet measured are marked MEASURING — nothing here is estimated silently.
2,729PRs in the corpus (435 labeled)
3,786labeled review findings
1,609review rounds parsed
286revert commits scanned
The brief proposed changes on the strength of published research. This doc tests those premises against our own repo. One of the brief's load-bearing assumptions is refuted, and the biggest cost line turns out not to be review at all.
1 · Claim → verdict
| # | Claim | Verdict | The metric that decides it |
|---|
| H1 | AI review gets noisier (higher false-positive rate) as diffs get bigger |
REFUTED — sign reversed |
FP rate falls monotonically 23.4% (<50 lines) → 6.1% (>1500); 95% CIs 13.6–37.2% vs 4.8–7.8% do not overlap. Worst-case ranking (all UNCLEAR counted as FP) is identical: 62.1% → 29.1%. Mechanism is severity mix — blocking share climbs 8.5% → 25.1%. |
| H2 | Review yield rises with diff size |
CONFIRMED |
Findings/PR 2.21 → 14.55 (6.58×); %-of-PRs-with-any 76.8% → 100.0%. On the blocking-only cut it steepens to 13.6×, matching the published curve's shape. |
| H3 | Extra rounds re-litigate rather than converge |
CONFIRMED (both halves) |
New findings/round 4.14 → 2.03 → plateau ~1.5; from round 2 on, 43–51% of everything surfaced is a repeat. 51.0% of all new findings arrive after round 1. 🟡 stays the modal verdict out to round 16 while 🔴 vanishes after ~round 5. |
| H4 | Lead time is dominated by review round-trips |
CONFIRMED — but the driver is idle time, not review |
80.7% of lead time is >60-minute dead air (no commit, comment, review or merge). Open→first verdict is p50 1.0 minute and only 9.6% of lead time. p50 lead time 62.5 min, p90 752.1 min. |
| H5 | Big PRs ship more defects |
CONFIRMED |
Reverts per 100 merged 1.81 → 9.92 (5.5×), monotone across all five size buckets under both attribution methods (primary method: 3.1 → 17.94). Reverts are a lower bound on defects. |
| H6 | Abandoned PRs re-land and pay for review twice |
CONFIRMED that it happens; magnitude INCONCLUSIVE |
120 of 163 closed-unmerged PRs (73.6%) have an identifiable successor; 98,019 lines were reviewed then thrown away. Only 6 pairs fall in the labeled sample (15 rounds, 40 findings), so any total is an extrapolation off n=6. Treat 73.6% as an upper bound. |
| BONUS | Un-gating the test lane (tests run in parallel with review) wastes runs |
Waste is small — 6.0% |
Only 163 of 2,704 decided PRs never merge, and a wasted parallel run requires the PR to die, not to go red. Round-1 verdicts: 23.9% 🔴 / 57.7% 🟡 / 16.4% 🟢. Range 3.7–23.9%, best estimate 6.0%. |
In plain terms
We thought the reviewer got sloppier on big changes. It's the opposite: on tiny changes about one finding in four gets waved away as not worth doing, versus about one in sixteen on the biggest changes — small changes attract nitpicks, big ones attract real bugs. And waiting on the reviewer is not what makes shipping slow: the reviewer answers in about a minute. Four out of every five hours a change spends open, nobody and nothing is touching it.
Caveats carried forward. 27.7% of findings are UNCLEAR, so every fate-based rate is reported twice (excluding UNCLEAR, and worst-case). 1,028 bot comments carried no parseable verdict marker, so round counts are lower bounds. H1–H4 and the bonus are sample-based (435 PRs); H5 and H6 use the full corpus.
2 · Cost ledger — the path to −50% Actions minutes
~380k minAug projected minutes/mo ESTIMATE — linear from Aug 1–8
~$2,022Aug projected gross/mo (~$1,723 net billed)
139 → 296minutes per PR, Jul → Aug (+113%)
71%of Actions minutes are ci.yml alone
The 50,000-minute included pool is exhausted every month, so every marginal minute is billed at full rate ($0.006/min x64, $0.005/min ARM). Daily burn is flat month over month — but PR throughput fell from 1,451 (18 days of July) to 333 (9 days of August), so the cost per PR more than doubled. The fixed cost (12-shard suite, nightlies, harnesses) did not scale down with volume.
Where the money actually is (August window, 2026-08-01..09)
| Workflow | Share of repo minutes | Est. minutes | Runs (ok / fail / cancelled) |
|---|
ci.yml | 71.4% | 70,322 | 2,180 / 306 / 1,964 |
claude.yml | 7.2% | 7,058 | 728 / 18 / 2 |
auto-merge-all.yml | 4.5% | 4,384 | 2,908 / 29 / 607 |
claude-code-review.yml | 4.5% | 4,377 | 477 / 120 / 11 |
semgrep.yml | 2.8% | 2,745 | 1,404 / 18 / 70 |
| everything else (20+ workflows) | 9.6% | ~9,545 | — |
Cancellation churn is the single largest identified waste line. ci.yml cancelled 1,964 runs against 2,180 successes — a cancelled run for every 1.1 that finished (44% of all its runs; 47% measured against successes). Across all workflows, cancelled runs cost ~$309 per 30 days and failed runs another ~$140. Cancelled runs are modeled at 40% of a full run — an estimate; run counts and billed totals are hard.
Two things the audit cleared, so we stop looking there: docs-only path filtering works (every docs-fence run is a zero-billable job-level skip), and manual re-runs are ~1% of runs. All large fan-out jobs already carry path filters. The remaining cost is ci.yml volume, not missing filters.
The ledger, itemized by PR
One of the six has landed (#5583, merged 05a87b02); the other five are still open. Effects measured overnight are stated as actuals below; only two lines remain unmeasured.
| PR | What it does | Cost line it attacks | Measured effect | State |
|---|
| #5581 | Verdict watchdog + dispatch — pages #alerts when a PR waits on a review verdict that never lands | H4 dead air (80.7% of lead time), not minutes | Latency only; no minutes saving claimed | Open |
| #5582 | Incremental re-review — round 2+ reads only the delta since the last verdict | Reviewer lanes (claude*.yml ≈ 11.6% of minutes) + H3 carryover | 1,374 → 85 lines of review input on the PR #5573 replay (−94% input). Minutes saved on the reviewer lane: MEASURING — will update | Open [HOLD] |
| #5583 | ci debounce of PR bursts + fail-fast on the 12-shard lane | The 1,964 cancelled ci.yml runs (~16,886 est. min in 9 days) and 306 failures | Mechanism proven. PR-only preflight (40s debounce + tip check): a double push 14s apart cost 1 job start instead of ~7 (run 31291953016 cancelled in-graph, 31291959398 succeeded). Deploy path provably unchanged. New preflight cost ~$38/mo against the ~$310/mo churn class it attacks. Monthly savings accruing over the week — the dollar delta is not yet banked. | Merged 05a87b02 |
| #5587 | Security fix — the reviewer prompt must not be sourced from the PR head (prompt-injection via branch contents) | None (correctness/security) | n/a | Open [HOLD] |
| #5588 | Workflow hygiene batch — least-privilege permissions, concurrency groups, trigger scope | Duplicate/superseded runs feeding the cancellation line; audit security flags | MEASURING — will update | Open |
| #5589 | Guard union — one vitest process for the four always-run guard pins | The affected lane's fixed floor | Autopsy: 4 always-run guard steps = 74% of the 7.6-minute median lane. MEASURED across three PR runs: guard block 358s → 296–319s (−11% to −17%, ~45s/run median); lane job 7m02s → ~6m. The 4-core ARM runner in the original target was deliberately dropped — no such runner label exists in the org, and jobs requesting an unprovisioned label queue forever. Provisioning it is an org-admin (Fede) decision, and that is where the rest of the ≤3.8-min target lives. | Open |
Does this reach −50%? Not on today's evidence. The lines in the ledger's line of fire — ci.yml cancellation churn (~17k min/9d), the reviewer lanes (~11.4k min/9d) and the guard floor — together account for roughly a third of the August bill (the share is hard; the recoverable fraction of it is still accruing). #5583 has proven its mechanism but its dollars arrive over the coming week, and #5589's remaining minute is blocked on a runner label only an org admin can provision. Reaching −50% needs one structural decision on top, which is what section 5 asks for.
In plain terms
Almost three quarters of the CI bill comes from one job, and nearly half the times that job runs, someone pushes again and it gets thrown away mid-flight. The fix for that has landed and demonstrably works — one double-push that used to start seven jobs now starts one — but the savings show up on next month's bill, not tonight. The rest needs a decision about what we're willing to stop running on every push.
Overnight incidents (2026-08-08 → 09)
| Incident | What happened | Root cause | Status |
|---|
| Production served a stale build for ~3h — ceiling (a), build-container OOM |
After #5577 added ~385MB to the graph, page-data collection ran 3 full-graph workers and peaked at 6,899MB against an 8GB build container. Production Promote failed and prod kept serving the previous build. |
Unbounded build parallelism: three concurrent full-graph workers each holding the enlarged module graph. |
Fixed by #5594 (experimental.cpus: 2) — peak 3,751MB, −46%. |
| Same outage — ceiling (b), 250MB function limit |
api/vendor-quotes/poll reached 250.07MB uncompressed, just past Vercel's 250MB limit. |
The googleapis umbrella package (195MB, eagerly requires ~400 APIs) rode in through file tracing. |
Fixed by #5595 — switched to @googleapis/gmail; function 64.4 → 30.1MB. #5577 was not reverted; prod verified serving main tip 6001c0f61d, Promote green on run 31301272570. Second time the 250MB limit has wedged prod (prior: 2026-08-02). |
| Verdict-gated Unit Tests re-park PRs |
The Unit Tests lane requires a fresh verdict per commit SHA, so every auto-update against a busy main sends the PR back to the start of review. 5+ extra review cycles across #5579 / #5580 / #5595 in one night, each ~3–5 min of review compute plus wall-clock. On the outage above the deferred lane alone cost ~1.5 hours of wall clock (rebase → re-verdict → 12-shard cycles) — while prod was serving a stale build. |
Gating tests on a per-SHA review verdict couples test eligibility to main's merge rate — a defect of the gate itself, not of any PR. |
Open — directly strengthens Decision 2 (un-gate the test lane). |
| Reviewer token outages |
Outages at ~03:0x, 04:06–04:16, and a further race at 06:08 even with the pool on a 5-minute rotation. |
Confirmed: ccswitch's background re-mint invalidates the pooled OAuth access tokens between rotations. Rotating faster cannot close the race — the invalidation is the mechanism. |
Open — see Decision 4. |
| Live ElevenLabs sync verified (not a failure) |
After the transfer-briefing merges, the sync-specialists main run 31299153164 synced transfer destinations and tool descriptions, and its post-sync live drift check passed. |
n/a — this is the check that catches a repo change never reaching the live agent. |
Verified end-to-end on the live agent |
In plain terms
Three things went wrong overnight and one went right. The live site quietly stayed on yesterday's version for about three hours because a deploy hit two separate ceilings at once: the machine that builds the app ran out of memory, and one API endpoint grew past the largest size the platform will accept — a giant Google library that pulls in four hundred APIs to use one of them. Both are fixed, and the endpoint is now less than half its old size. Notably it is the second time that size limit has stopped a deploy in a week. Separately, the rule that tests may only run after a fresh review means that any time someone else merges, the change gets sent back to the start of the queue — that happened five-plus times in one night. The reviewer also kept losing its login, and we now know exactly why: a background tool re-mints the token and silently kills the one in use. The good news: the voice agent's live settings were confirmed to match what's in the code.
3 · Config audit
Actions in use
321 uses: references across the workflow directory. The top ten: actions/checkout@v4 (87), actions/setup-node@v4 (79), aws-actions/configure-aws-credentials@v4 (26), actions/upload-artifact@v4 (25), actions/cache@v4 (22), local ./.github/actions/lambda-smoke (21), local lambda-role-parity-check (13), actions/github-script@v7 (10), actions/cache/restore@v4 (9), anthropics/claude-code-action@v1 (5).
Security flags
| Flag | Evidence | Why it matters |
|---|
| No SHA pinning anywhere | 0 of 321 uses: references are pinned to a commit SHA — all are floating tags (@v4, @v1) | A compromised or retagged third-party action executes in a job that holds AWS keys and the automerge token. This is the tj-actions/changed-files failure mode. |
PROD_COOKIES_JSON is a repo-wide secret | Present in the repository Actions secret list alongside AWS_SECRET_ACCESS_KEY, AUTOMERGE_TOKEN, SMOKE_* credentials | A live authenticated production session is readable by any workflow in the repo rather than scoped to the one environment that needs it. |
| Over-privileged status-dashboard app | Audit finding: the dashboard's GitHub App carries write scopes it never exercises | A read-only reporting surface holds write authority on the repo. Downgrade to read scopes. |
| 3,986 stale branches | 3,990 remote heads on origin at time of writing | Not a security hole by itself, but it is the surface every branch-triggered workflow and every similarity-matching tool has to reason about — and it is where H6's abandoned work is parked. |
In plain terms
Every outside tool our build pulls in is fetched by nickname, not by exact version — if someone hijacks one of those nicknames, their code runs next to our AWS keys. And a logged-in production session is stored where any build job in the repo can read it. Neither has caused an incident; both are one bad day away from one.
4 · The real-time event system
H4 says 80.7% of lead time is dead air. A meaningful slice of that is the agent not knowing anything happened: a session finishes, stops, and only learns about a review verdict when someone next polls. The fix now running is a real-time event path:
gh webhook forward → local receiver → asyncRewake hook. GitHub's CLI forwards repository webhook events to a receiver running on the machine (kept alive by a LaunchAgent). A blocking wait script is registered as a Claude Code hook with asyncRewake: true; when a PR event lands, the hook returns and wakes the session with the event, instead of the session sleeping until a human pokes it.
Registered on both SessionStart and Stop, so a session is re-woken whether it is idle or has just finished. Verified in place: ~/.claude/scripts/gh-event-forward.sh, ~/.claude/scripts/pr-event-wait.sh, ~/Library/LaunchAgents/com.propflow.gh-event-forward.plist, and the hook block in ~/.claude/settings.json.
It is generic. Nothing in it is PropFlow-specific: two scripts, one plist, one hook block. Any repo and any Claude Code install can adopt it as-is — which makes it the cheapest thing in this doc to share outward.
Real-time push: before/after benchmark
Metric: verdict → first-reaction latency — minutes from a claude review verdict landing on a fede-authored PR (bot verdict comment, or a submitted APPROVED/CHANGES_REQUESTED review) to the first subsequent actor response on that PR: a new commit, a non-bot comment, or the merge. That interval is the dead air the push system targets.
| Window | n verdicts | p50 | p90 | mean | max |
|---|
| BEFORE — 2026-07-15 → 08-09 04:00 UTC, 830 PRs | 1,921 | 8.5 min | 42.6 min | 40.5 min | 9,303 min (6.5 d) |
| AFTER — 08-09 04:00 → 15:11 UTC, 22 PRs | 68 | 9.3 min | 29.0 min | 15.1 min | 160 min |
| AFTER, excluding the two outage windows (04:06–04:16, 12:09–12:12 UTC) | 66 | 9.3 min | 29.6 min | 15.2 min | 160 min |
| BEFORE, merges excluded as reactions | 1,181 | 9.5 min | 64.6 min | 40.5 min | 2,408 min |
| AFTER, merges excluded as reactions | 58 | 9.3 min | 32.5 min | 16.1 min | 160 min |
The honest read: the median did not move; the tail collapsed. p50 is flat (8.5 → 9.3 min) — a verdict that was going to be picked up promptly was already being picked up promptly. What changed is everything behind it: p90 42.6 → 29.0 min (−32%), mean 40.5 → 15.1 min (−63%), and the worst case 6.5 days → 2.7 hours. The push path does not make a working agent faster; it removes the case where nobody was watching at all.
Reaction mix, BEFORE: 790 commits, 740 merges, 391 human/agent comments. Merges are counted as reactions and labelled here rather than dropped — excluding them (rows 4–5) moves the p90 gap wider, not narrower, so the headline is not a merge artifact. Verdicts that never drew any reaction are excluded: 22 dropped BEFORE, 1 dropped AFTER.
Method: BEFORE from the harness corpus review-harness/data/prs.jsonl (2,729 PRs, Jul 1 – Aug 8) using the same verdict parser as the round analysis; AFTER pulled fresh from the GitHub API for every fede-authored PR with activity after the 2026-08-09 04:00 UTC cutover (#5577–#5600). Per-verdict rows: review-harness/data/before-latency.json, data/after-latency.json; script review-harness/bench_latency.py.
Caveats, stated plainly. (1) The after-sample is one abnormal night — a token outage and a production incident inflated some gaps, which is why the outage-excluded variant is shown; it barely moves, so the outage is not carrying the result. (2) n=68 against n=1,921: the after figures are indicative, not settled. (3) Faster reaction also reflects agents simply being active that night, not the push system alone — this measures the two together and cannot separate them. (4) Commit reaction times use committedDate, a proxy for push time. (5) On PR #5600 the bot posted both a verdict comment and a formal APPROVED review minutes apart; both count, which slightly over-weights that one PR. MEASURING — a full week of post-cutover PRs will replace the AFTER row
In plain terms
Before, a finished agent sat there until somebody checked on it. Now GitHub taps it on the shoulder the moment a review comes back, and it picks the work straight back up. It's four small files — any team using Claude Code can copy it.
5 · Decisions
Four choices. Each is multiple-choice with a recommendation; picking A on all four is a coherent plan on its own.
Decision 1 — What do we do about small-PR noise, now that H1 is refuted?
OPTION A — RECOMMENDED
Invert the brief's confidence gate: suppress low-severity findings on small diffs, keep full depth on large ones. The FP rate on <50-line PRs is 23.4% against 6.1% on >1500, and blocking share climbs 8.5% → 25.1% with size. Depth escalates on risk paths (auth, payments, migrations, PMS writes), never on size alone — size triggers a split instead.
OPTION BGlobal confidence gate — only high-severity findings block, at every size. Simpler, but throws away the real yield on large PRs (3.66 blocking findings per PR in the top bucket).
OPTION CChange nothing here and spend the effort on rounds and cost instead. Defensible: 9.4% overall FP is below the 10% "trusted" threshold. Cost: small PRs keep paying a full round to dismiss a nit.
Decision 2 — Do we un-gate the test lane?
OPTION A — RECOMMENDED
Un-gate: tests run in parallel with review. Measured waste is 6.0% of runs (range 3.7–23.9%), because only 163 of 2,704 decided PRs ever die. In exchange 94% of PRs get their test signal a full review cycle earlier — against a mean of 3+ rounds and each round past the third roughly doubling median lead time. Tonight's gate-loop defect settles it: because the lane demands a fresh verdict per SHA, every auto-update against a busy main re-parks the PR — 5+ extra review cycles in one night, and ~1.5h of wall clock on the outage fix while prod served a stale build. #5583's debounce is already merged, so the parallel runs won't feed the cancellation line.
OPTION BUn-gate only PRs under a size/risk threshold, keep the gate on large or risk-path PRs. Captures most of the latency win at lower run volume; more machinery to maintain.
OPTION CKeep the gate. Cheapest in minutes, and the honest reading if we decide the −50% goal outranks lead time. Cost: every PR keeps paying a full review cycle before it learns whether it even compiles.
Decision 3 — How far do we push for −50% Actions minutes?
OPTION A — RECOMMENDED
Finish landing the ledger, let #5583's savings accrue, then decide. #5583 is merged with its mechanism proven; merge #5581/#5588/#5589 under the standing grant, hold #5582 and #5587 for review, and re-publish this section with a real month-over-month minute delta. Two open items need Fede specifically: provisioning a 4-core ARM runner label (org-admin only — it is where the rest of #5589's target lives) and the token decision above. Committing to a structural cut before #5583's dollars land would be guessing.
OPTION BAdd a structural cut now: stop running the full 12-shard suite on every main merge, moving it to a merge queue or an hourly batch. Almost certainly gets to −50% by itself, given ci.yml is 71% of minutes. Cost: a broken main is detected in up to an hour rather than immediately.
OPTION CAccept the bill. ~$1,723/mo net is real but small against engineering throughput, and August's spike is partly a volume artifact (throughput fell 4×; fixed cost did not). Fix only the waste lines and drop the −50% target.
Decision 4 — How do we stop the reviewer losing its token?
Root cause is now confirmed, not suspected: ccswitch's background re-mint invalidates the pooled OAuth access tokens between rotations. Three outages last night (~03:0x, 04:06–04:16, and a race at 06:08 with a 5-minute rotation already in place) — so rotating faster does not fix it.
OPTION A — RECOMMENDED
Move the reviewer to a long-lived token (claude setup-token, 1-year). It is not minted by ccswitch, so the invalidation path disappears entirely rather than being raced. Trade-off: a year-lived credential in repo secrets — pair it with the #5588 least-privilege pass and a calendared rotation.
OPTION BIsolate the pool from ccswitch — mint reviewer tokens under a separate credential store the switcher never touches. Keeps token lifetimes short; costs a second credential path to maintain, and any future tool that re-mints re-opens the same class of bug.
OPTION CKeep short rotation and lean on #5581's watchdog to page when a verdict never lands. Cheapest, and detection is already built — but it accepts recurring outages as normal and each one stalls every open PR.
In plain terms
Four calls to make. One: stop the reviewer nitpicking tiny changes, and leave it at full strength on big ones. Two: start running tests at the same time as review — about six runs in a hundred get wasted, and the other ninety-four learn a whole cycle sooner. Three: land the changes already written, see what they actually save, and only then decide whether to stop running the full test suite on every merge. Four: give the reviewer a login that lasts a year, because the tool that refreshes its current one keeps quietly cancelling it mid-review.
How to reproduce
python3 review-harness/analyze/analyze.py → data/analysis.json; python3 review-harness/analyze/gen_report.py → analyze/REPORT.md. Reads only local data/*.jsonl and data/*.json; no network. Cost figures come from data/actions-cost.json, whose billed totals, run counts and conclusion counts are hard and whose per-workflow splits are ±20% estimates calibrated to the billing API (0.778 July, 0.796 August).