Overnight system review: page-speed push, 2026-09-28
A read-only look at one driver session and its ~12 helper lanes over about 14 hours. It covers what works, whether the guardrails are right, whether we're working on the right thing, whether sessions drift, and where the time went.
Labels: measured means read from logs or GitHub today. estimated means inferred from timestamps. reported means someone else's figure that I did not re-check.
In one screen
- Lots of careful work shipped. About 200 PRs merged overnight, median about 200 lines and about 30 minutes to merge; reviews caught real bugs, including an access gap in four pages.
- The page-speed list didn't move (18 open). For much of the night half the open rows had no owner, most finished fixes were never recorded on their row, and one PR waiting for a human merge (#12791) blocked 11 rows.
- Sessions didn't wander off chasing bugs on their own. Scope changes came from the driver re-tasking idle lanes, often onto non-page-speed work. Found bugs mostly got parked as board rows, but nothing in the lane rules requires that.
- The plumbing cost the most time. About 2 hours with no deploys (each test run on main was cancelled by the next merge), and about 46 minutes of broken main from a check that only runs after merge.
- Top 3 changes: (1) every task and PR names the board row it moves, and records it there at merge; (2) a twice-daily human-merge window, biggest blocker first; (3) let main's test runs finish, and run after-merge-only checks before merge.
Details below. Every number is labelled measured, estimated or reported.
Summary (full)
- The machine ships a lot, and ships carefully. At least 200 PRs merged repo-wide since 02:00Z, with a median of 197 lines and a median of 29 minutes from open to merge measured. The review bot and the lanes' own self-checks caught real defects, including a security hole.
- The page-speed board did not move. It started and ended the night at 18 open rows. At 13:30Z, 20 of the 21 page-speed merges listed then were not recorded against any board row measured.
- The main reasons are ownership and a human-merge bottleneck, not effort. 9 of the 19 open rows had no lane at all until 09:49Z measured. One PR waiting for a human merge (#12791) was the named blocker on 11 of the 18 rows measured.
- Sessions did not drift on their own to chase bugs. Every scope change I found was the driver re-tasking an idle lane, often onto work outside page speed. Found bugs were mostly parked correctly.
- The biggest time losses were in the plumbing. Production went about 1 h 53 m without a deploy because each main test run was cancelled by the next merge. Main was also broken for about 46 minutes by a check that only runs after merge measured.
Output by driver, last ~10 hours
| Driver | PRs opened | PRs merged | Focus |
|---|---|---|---|
| Driver 1 (page speed) | 78 | 67 | Page-speed rows, plus Found rows, Phase 9 triage and the staging-copy incident |
| Driver 2 | 34 | 30 | Earlier phases, review tooling |
| Driver 3 | 21 | 15 | Cross-company access fixes |
| Driver 4 | 1 | 1 | β |
The PR counts are reported by driver 1 and were not re-counted. What I checked myself measured: GitHub shows at least 200 merges since 02:00Z (the query stops at 200). Of those, 132 went through the automatic merge guard's account and 68 through the drivers' account. A few of the guard-account merges may be human hand-merges.
Strongest traits of the successful sessions
- They say "I was wrong" quickly and in writing. Lane G withdrew its own "real visitor" speed figure when it found those visits were mostly bots and older code. Only one real customer visit was tagged on each of the two pages, so the rows stay open until 20 or more accrue.
- They check their own measuring tools. Lane M found a bug in its own occupancy probe and re-ran it. The shared production-read log shows the driver's own waiter jumping the queue by matching an old "done" line, and the log corrects it within the same minute (16:24:58Z) measured.
- They ask one pointed question, then fan it out. The driver's question on #13169 exposed a permission check being cached for up to 90 seconds. Driver 3 then audited the similar pages and found the same hole in three more (#13176, #13177, #13178). All four fixes have tests that fail on the old code.
- They stop and ask at the edge of the brief. Lane T paused on a caching design choice instead of guessing, and the driver answered "Good catch, and thanks for stopping." Lane REV's plan to change a write path was stopped in favour of "measure first, read-side only."
- They keep production decisions with a person. Every production write overnight waited for Gera's explicit OK, and each was logged with an undo step. That covered the staging-copy cap, the memory increase and the 15,769-row email-log fill.
- They slice small. The median merged PR is 197 lines, under the 300-line target measured. 65 of the 200 were over it.
Guardrails
Earned their keep
- The review bot. It blocked #13156 (π΄) because a cache would have shown a stale live grade; that was fixed and green again in 10 minutes. It flagged a stale header on #13148 (π‘), green again in 8 minutes measured, from PR comments.
- Human merge on access code. Every permission and sign-in change waited for a person. That caught the "viewer gets two opposite answers" problem on #12979 before it shipped.
- The one-at-a-time production-read slot. There were 65 read-only production runs overnight, and none of them caused throttling measured, from the slot log.
- Tests that must fail on the old code. They proved the four access fixes and several speed fixes were real.
Cost time without catching much
- Advisory checks that get cancelled. Cancelled paperwork checks (on the PR description) kept PRs from counting as "all checks complete" and had to be re-run by hand. At least four incidents were logged today measured. I didn't measure the time they cost.
- Driver read-receipts sent as chat. The driver sent 192 messages to six lanes in about 14 hours measured. 16 of lane T's 43 messages were "I've read your diff at this version." That's careful work, but it spends the driver's attention on something a file or label could carry.
Gaps that let things through
- A rule check that runs only after merge. #13135 passed both plan review and code review (π’) and merged, then broke main. The check that forbids a hand-written "back" link runs only in the full post-merge test run. Main was red for about 46 minutes, from the 13:51Z merge to the 14:37Z fix measured. The 10:10Z red main (#13059) had the same shape.
- Merged before anyone read it. 12 of the 144 diff-read receipts written today say the guard merged the PR before the driver read it measured. On one of them (#12982), the driver's later read also missed the bug the bot missed: the page fetched its data twice. The after-the-fact read is not a reliable safety net.
- A plan verdict ignored. #12968 merged past a "contradicts the plan" verdict the driver had not weighed. That lesson is now written into the lane rules.
Working on the right thing
- Many merges never reached the board. At 13:30Z, 20 of 21 page-speed merges had no reference on any board row, because lanes hadn't sent their receipts to the board clerk measured. A fix that isn't recorded can't close a row.
- Half the open rows had no owner for most of the night. 9 of the 19 open page-speed rows were unowned until 09:49Z measured, which is about 7 hours estimated. Meanwhile lanes worked on Phase 9 triage, Found-along-the-way rows, reviving idle PRs and the atlas screens.
- Rows need things lanes can't produce. They need real-visitor timings, and almost no human visits have been tagged yet. They need a human merge: #12791 was approved at 07:57Z and merged only after Gera's go-ahead at about 16:25Z. Or they need a decision from Gera. Rows that were "one recorded receipt away from closing": zero measured.
- Is there a reviewer of direction? Partly. Each PR gets a plan review (conforms, diverges or contradicts) against the written design. But nothing checks that a PR names the board row and criterion it is meant to move, or that every open row has an owner. The driver did that by hand, once, at 09:49Z.
Do vague instructions drift? The bug-detour question
Finding: I saw no self-driven drift. I read every message the driver sent to six lanes. Each scope change came from the driver, usually as "while X waits on review, take this." None came from a lane that found a bug and wandered off.
- Lane T started on "phone numbers follow the company pick." The driver then gave it Phase 9 at 04:12Z, an agents-page item at 05:40Z and the portfolio report at 09:49Z. About 1.5 of its ~14 hours went on the original ask estimated. The rest was driver-assigned, and much of it wasn't page speed.
- Lane R started on the email log and #12735 and stayed there until about 08:29Z, about 5.8 hours estimated. It was then sent to the next page-speed row (the admin developer page), which was on-plan.
- Lane REV started on "revive idle PRs" and did that until about 09:48Z, about 4 hours estimated, with one extra item at 06:01Z. It then got the leasing row. When it tried to widen into a write-path change, the driver stopped it.
- Driver 3's permission-cache audit was a detour from its queue, but the driver's question prompted it and it found three real holes. That's the kind of detour worth taking.
- One case fits Gera's worry. Lane M, the page-speed measurement lane, spent roughly 05:50Z to 10:10Z on the staging-copy job that was overloading the production database estimated. It was a genuine emergency and Gera approved each step, but page-speed measurement paused meanwhile.
Is there a way to park a found bug instead of fixing it inline? Yes: "Found along the way" board rows, and they were used. The board clerk filed rows for read-count tracking and for a deploy-version bug instead of anyone fixing them on the spot. There are two weaknesses:
- The lane rules don't tell a lane to park a found bug. That discipline lives only in the driver.
- The parking lot isn't drained. It holds 164 open rows, and 38 of them were already fixed on main but never closed measured, from the morning triage.
Efficiency: where the time went
- No deploys for about 1 h 53 m measured. From 11:43Z, 29 main test runs in a row were cancelled by the next merge, because merges landed faster than one full run takes. Nothing merged in that window reached customers until 13:36Z.
- Broken main for about 46 minutes from #13135 measured. There was also an earlier red at 10:10Z (#13059, fixed by #13087).
- The production-read slot was not a bottleneck. It was busy about 196 of about 830 minutes, across 65 runs measured. The losses were hand-off delays (lane R started 45 minutes after its turn came up measured) and two log-reading slips: one matched a start or done marker in the middle of a line, and the other caused the queue jump.
- Review rounds were quick when the findings were real: 8 to 10 minutes from verdict back to green on the two PRs checked measured. Re-reviews after a catch-up with main (no code change) appear often in the log, but I didn't measure their total.
- Waiting for a human merge was the longest single delay. #12791 sat approved for about 8.5 hours estimated, from report timestamps, and it blocked 11 rows. The morning report listed 10 or more other PRs as "ready for your merge."
- Lanes spent roughly a third of their hours outside page speed estimated, from the re-tasking timestamps above.
Top 3 changes, ranked by payoff
- Make every lane item and PR name the board row and criterion it moves, and have the lane send the board receipt at merge. Before handing any lane off-phase work, the driver re-lists open rows that have no owner. This alone would have put 20 unrecorded merges on the board and staffed 9 empty rows about 7 hours sooner.
- Give Gera a twice-daily human-merge window, with the single biggest blocker (like #12791) at the top. One waiting merge held back 11 of 18 rows for most of a day.
- Stop main's test runs cancelling each other, and run the post-merge-only rule checks on any PR that touches their files. Letting a run on main finish instead of being replaced removes the ~2-hour deploy freeze. Running those checks before merge removes red mains like the ~46 minutes from #13135.
Plain English: the team shipped a lot of careful, well-checked work overnight, but the page-speed board didn't move because half the rows had no one assigned, most finished fixes were never recorded against their row, and one waiting hand-merge blocked most of the rest.