Proposed — pending review

AI code review: keeping quality high while cutting lead time to change

Research brief + proposed changes · 2026-08-09 · sources cited per finding; unsourced items marked JUDGMENT

Why now

317 → 654

median merged-PR lines, July → Aug (2×)

0.4h → 1.0h

median time-to-merge (p90 now 8.5h)

1 → 4

median review threads per PR (4×)

8 rounds

review rounds on the worst August PR

August also produced two huge-PR-closed-and-relanded-smaller cycles (21k and 3.6k lines), and a review-bot token outage that silently stalled every open PR for ~2 hours (fixed 2026-08-09: token pool rebuilt on a 2h rotation with review-sized health probes). All figures above are measured from the repo's own PR history.

What the research says

1 · LLM review precision collapses as diffs grow. A comparative evaluation across diff-size buckets found F1 falling steeply with size — and the loss is precision-led: big diffs produce more false findings, not just missed ones. The largest measured bucket tops out at 600 lines; our median PR is past it. The authors' conclusion: constrain input size rather than scale the model.

arxiv.org/pdf/2606.15689 · confidence: high

2 · Anthropic's own review product uses fan-out + a verification pass, not one big rubric. Parallel finder agents, a separate verifier that filters false positives, severity ranking, depth auto-scaled to the change; reported <1% of findings marked incorrect. Yield curve: PRs >1,000 lines yield findings 84% of the time (avg 7.5); PRs <50 lines, 31% (avg 0.5) — the empirical case for a light pass on small changes.

claude.com/blog/code-review · confidence: high

3 · Small-CL discipline is the oldest result in the field. Google: 100 lines is a reasonable CL, 1,000 is too large; benefits include faster review, fewer defects, cheaper rejection, and the author staying unblocked. Stacked diffs are the mechanism that makes a size cap free for the author.

google.github.io/eng-practices · graphite.com/guides/stacked-diffs · confidence: high

4 · DORA 2025: AI moved the bottleneck from writing to reviewing. New fifth metric, Rework Rate; AI adoption correlates with delivery instability driven by volume; the stated amplifier is small batches. Secondary telemetry: PR size +51%, review time +441% in 2026 data. Our July→August shift is a textbook instance.

faros.ai (secondary) · confidence: medium-high framing, medium numbers

5 · False positives price differently under auto-merge. Practitioner thresholds: <10% FP → findings trusted; >30% → triaged with suspicion. JUDGMENT Because our merges gate on the bot's verdict, a false finding can't be shrugged off — it must be argued down, so every FP converts into a full review round. Our 8-round PR is the FP cliff expressed as latency.

cubic.dev · confidence: medium

6 · Path-scoped rubrics beat monoliths. GitHub's Copilot review now layers short path-scoped instruction files (applyTo: globs); guidance is explicitly anti-monolith — shorter files are more reliably honored. Our single large rubric maximizes input on every PR regardless of risk, the exact condition finding 1 warns about.

github.blog changelog · confidence: high

7 · A bot that never posts is invisible without a dead-man's switch. The standard pattern: jobs ping only on genuine success; an external watcher alerts on the missing ping. Our August stalls (no verdict marker → auto-merge silently held) are exactly the named failure mode. Pair with timeout-minutes and workflow_dispatch.

dev.to guide · confidence: high pattern

Proposed changes

Quick wins (days)

#	Change	Attacks
Q1	Incremental re-review — after round 1, review only the delta since the last-reviewed SHA, carrying prior findings as context. Store reviewed-SHA + verdict idempotently.	Rounds, size, cost at once — highest leverage
Q2	Dead-man's switch — scheduled watcher alerts #alerts when an open PR's head has no verdict after N minutes; `timeout-minutes` on the review job.	Silent stalls (tonight's outage class)
Q3	Verdict-absence must read as "pending", never as consent — required check that starts pending, not absent.	Auto-merge safety
Q4	Split the rubric into a short universal core + path-scoped fragments (API, DB/migrations, auth/PII, UI, tests).	Prompt size → precision
Q5	Risk-tiered depth + skip-list — lightweight pass for small/low-risk paths; deep review + human sign-off for auth, payments, migrations, PMS writes, saga code. Escalate depth on risk, not size — size should trigger a split instead.	Wasted depth on low-risk PRs
Q6	Confidence gate — only high-confidence, high-severity findings block the merge; the rest post as non-blocking comments.	FP-driven review rounds

Structural (weeks)

#	Change	Attacks
S1	PR size budget — warn at 400 substantive lines, block at ~800 with an override label.	The root cause of the August regression
S2	Stacked PRs so the size cap costs no author throughput.	Makes S1 sustainable at 50 PRs/wk
S3	Fan-out + verify architecture — parallel specialized sub-reviews, then a verifier that must confirm each finding against the repo before it can block.	False positives (the <1% architecture)
S4	Verdict caching by file-set × content hash — rebases and no-op merges from main never re-review.	Token pressure → infra flakiness
S5	5% sampled human audit of auto-merged PRs; log agreement, feed back into rubric.	Keeps the auto-merge licence honest
S6	FP corpus feedback — findings argued down in review rounds become "do not flag" rules in the relevant fragment.	Recurring false findings

Decision needed

One choice from Fede sets the scope of the first implementation pass:

OPTION A — RECOMMENDED

Ship all six quick wins (Q1–Q6) as one focused pass, defer structural. Q1 + Q6 attack the round-count directly, Q2/Q3 close the outage class we just lived through, Q4/Q5 shrink prompt input. No workflow-culture change required; measurable within a week against the August baseline.

OPTION B

Quick wins + S1/S2 (size budget + stacked PRs) now. Attacks the root cause immediately, but changes daily authoring workflow mid-launch-runway; higher adjustment cost.

OPTION C

Q2/Q3 only (reliability), leave review shape alone until after Camellia/Yale launches. Cheapest, but leaves the precision and round-count problems compounding at current shipping volume.

What we'll measure

Lead time p50 + p90 (baseline: July 0.4h median; Aug 1.0h / 8.5h p90) — p90 is where the regression lives.
Rounds per PR and finding yield per round — healthy = yield collapses after round 1; flat yield means the bot is re-litigating, not converging.
False-positive rate (findings argued down ÷ total) — currently unmeasured; target <10%, treat >30% as an incident.
Verdict-missing rate + time-to-verdict p99 — the direct proof the stall class is closed.
PR size distribution (% over 400 / 800 lines) — the leading indicator; if it doesn't fall, nothing downstream will.
Rework rate (PRs closed-and-relanded) and change-failure rate as the quality guardrail.