An audit of the places where our automation works while the repo is small and breaks at a fixed number nobody chose, plus what is being done about each one.
2026-10-06 · propflowai, local-bin, agent-smith, propflow-docs · measured at main c8f04a1cf · Phase 1 of 2 (fixes tracked below)
Three merge-queue outages on 2026-10-06 were the same bug wearing three faces. A file list grew past 1 MB and a test crashed; a report grew past 64 KB and a pipe cut it; a test list grew past 128 KB and Linux refused to start the command. None of the code was wrong when it was written. The repo simply got bigger. This page lists every other place with the same shape, how close each is to its limit today, and what breaks when it goes.
How to read "headroom": 100% means nothing is in the bucket; 0% means it is full; negative means it is already overflowing and we are only getting away with it by luck. Anything under 50% on a queue-wide path gets fixed now.
| # | Where | What grows | Today / limit | Headroom | When it breaks | Fix |
|---|---|---|---|---|---|---|
| 1 | local-bin canon/drive-gate.py:643; propflowai queue-admission.sh:71, queue-reentry-guard.yml:358 | check runs on one commit (first page only) | 42–119 / 100 (PR heads); 66 (queue heads) | −19% to 58% | Driver gate says "GREEN, merge it" with checks it never saw. Queue sites are wording-only. | local-bin #445 (parked, reviewer down); propflowai #17189 baseline + guard |
| 2 | git ls-files captures (14 sites) | repo file listing vs Node's 1 MiB default | 1,048,692 / 1,048,576 B | −0.01% | Any guard test using it fails for every PR in the queue (the 14:18Z incident). | #17127 + #17160 merged; #17178 (every tracked code file) open |
| 3 | review-verdict-watchdog.yml:361; local-bin derive-parked:254 | open PRs vs --limit 100 | 78–97 / 100 | 3–22% | Silently skips the oldest (most stalled) PRs; the file's own comment claims the opposite order. | #17189; local-bin #445 |
| 4 | full-set-one-machine.sh:166 (npm test -- --exclude ×1,330) | guard union as one shell string (128 KiB cap) | 88,662 / 131,072 B | 32% | Every merge-queue train fails before vitest starts (exit 249). | #17100 open (Gera) |
| 5 | ci.yml:3066 GUARD_FILES env + :3095 npx vitest run $GUARD_FILES | guard manifest as one env string / one sh -c string | 58,152 (queue) – 75,362 (full) / 131,072 B | 42–56% | Guard pins shard cannot start; queue-wide. Growing ~27 KB/month (362 new drift tests in 30 days). | #17180 artifact transport + direct binary |
| 6 | post-merge-heavy.yml:222, one-machine-shadow.yml:347 | full guard union through npx's one-string sh -c | 75,362 / 131,072 B | 42% | Heavy lane goes blind on main; a heavy-lane-red issue holds deploys. | #17180 direct binary |
| 7 | …/actions/runs/<id>/jobs?per_page=100 — 13 propflowai sites, 6 local-bin, 1 agent-smith | jobs per ci.yml run (first page only) | 53 / 100 (19 → 53 top-level jobs in 30 days) | 47% | Merge verdict (auto-merge-all.yml:1826) and follow-up deciders judge an incomplete job list. 3 sites fail loud; 10 are silent. | #17189 (5 gate reads + guard, 12 baseline rows); local-bin #445; agent-smith #827 |
| 8 | branch-preview-domain.yml:222 | open PRs vs --limit 200 | 78–97 / 200 | 51% | Prunes live preview domains of PRs past the cut ("unrecoverable", per its own comment). | #17189 |
| 9 | pr-selection-run.sh:20, merge-group-affected.sh:53 CHANGED_FILES= | a PR's changed-file list as one env string | ~35,000 (683-file PR) / 131,072 B | 73% | A ~2,500-file PR cannot run its own test lane. | pass a file path; after #17100 |
| 10 | 6 scripts/ci scripts: process.exit(main()) after process.stdout.write | report size vs the 64 KiB pipe buffer | < 10 KB / 65,536 B today | > 85% | Output read by $(…) is cut mid-line, as #17050 was. | #17182 process.exitCode ×8; guard |
| 11 | CI git diff --name-only/--numstat/git log captures without maxBuffer; live-state-discipline.drift.test.ts:51 | diff / log / union size vs 1 MiB | ≤ 75,362 / 1,048,576 B | 93%+ | One CI script dies with ENOBUFS on a huge PR. | widen #17169's ratchet (part 2) |
A computer program often has a number baked in that says "I will hold this much and no more": a bucket size. Node holds at most 1 MB of a child command's output unless told otherwise. A Unix pipe holds 64 KB. Linux refuses to start a program if any single argument or environment string is longer than 128 KB. GitHub's API hands back at most 100 items per request. None of these are bugs. They are defaults.
The trap is that the data flowing through those buckets is tied to how big the company is: how many files are in the repo, how many tests we have, how many PRs are open, how many jobs a CI run has. All of those go up every week. So code that was correct in July silently fills its bucket in October, and the failure arrives on an ordinary Tuesday with no change that caused it. Worse, half of these buckets do not overflow loudly: the API just returns the first 100 and says nothing, the pipe just drops the tail of the report.
The fix pattern is always one of three, in order of preference: (1) stream, or write to a file and pass the file's path, so there is no bucket; (2) declare a bucket big enough that we will notice long before it fills, and fail loudly when it does; (3) never silently.
| Pattern | Limit | Found | Already covered |
|---|---|---|---|
A Whole-output capture without maxBuffer | 1 MiB (Node default) | ls-files: 15 sites. Other repo-scaled commands (git diff --name-only, --numstat, git log, git grep -l, union output) in CI code: 8, plus ~6 in tests. 553 child-process call sites total in propflowai; 77 already set a ceiling. | #17127 (1 site), #17160 merged (shared helpers/git-ls-files.ts + git-ls-files-maxbuffer.drift.test.ts, in the PR always-run set), #17178 open (every tracked code file); #17169 closed as superseded |
B Write then process.exit | 64 KiB (pipe buffer) | 6 in scripts/ci (guards-for-paths, guard-select, pr-file-collisions, heavy-proven-coverage, release-advance, stacked-base); 6 more in harness scripts (not CI gates) | #17050 (declared-test-reads.mjs) |
| C Unbounded list as one argv/env string | 128 KiB (MAX_ARG_STRLEN) — npx/npm run join the whole command into one sh -c string | 5 live npx vitest run $LIST sites (ci.yml:3095, post-merge-heavy.yml:222, one-machine-shadow.yml:214,347, cloud-test.yml:181); 3 in full-set-one-machine.sh; 2 env strings (GUARD_FILES, CHANGED_FILES); guards-for-paths.mjs:396 and find-culprit.ts:211 (bounded lists) | #17093 (reverted by the heavy lane as #17119), re-landed as #17100 (open) covering full-set-one-machine.sh only |
| D One page where all pages were meant | 100 items (GitHub REST), --limit N | /jobs: 13 propflowai + 6 local-bin + 1 agent-smith; /check-runs unfiltered: 3 propflowai + 1 local-bin; --limit on open PRs: 2 propflowai + 1 local-bin; --limit 6000 vs 14,105 merged PRs in reap-merged-worktrees.sh (fails safe: keeps worktrees) | gh-api-pagination.drift.test.ts covers comments/reviews/files only; 3 deciders already compare total_count and refuse a partial list |
| E Other fixed sizes | GitHub job output 1 MiB; env strings 128 KiB | Guard manifest as a job output: 75 KB of 1 MiB (fine). The same bytes as an env string: see #5. | — |
git ls-files -- src/ agents/ lambda/ scripts/ ':!**/*.md' | wc -c → 1,048,692 B at c8f04a1cf (23,321 tracked files, 1,269,290 B unfiltered).node scripts/ci/guard-union-files.mjs | wc -c → 75,362 B / 1,330 paths; --hard-only 58,152 B / 1,012; --ratchet-only 17,210 B / 318. As --exclude pairs: 88,662 B. Growth proxy: 362 *.drift.test.ts files added in the last 30 days.filter=latest): PR heads 42, 54, 84, 87, 119; queue head 66.gh pr list --limit 500 at 20:40Z. gh pr list returns newest first (17169, 17168, 17167).npm exec and npm run hand the whole command to sh -c as ONE string, so the per-argument limit applies to the entire file list. Calling node_modules/.bin/vitest directly makes each path its own argument (total limit ~2 MB). #17100's test proves it with a real npx on PATH.Each is a small PR off fresh main. Every guard is a *.drift.test.ts under src/__tests__/, which the PR "Always-run drift and decision tests" step runs on every pull request (the set is every *.drift.test.* plus every decide-*.test.*; scripts/ci/always-run-tests.mjs). Each guard ships with a sabotage control that proves it reds on a planted offender.
ci.yml guard-pins shards get the manifest as an artifact (the pattern selection.txt already uses) and call node_modules/.bin/vitest; post-merge-heavy.yml, one-machine-shadow.yml, cloud-test.yml call the binary. Guard: no npx vitest/npm test -- with a word-split list in workflows or scripts/ci. full-set-one-machine.sh is #17100's.scripts/ci scripts use process.exitCode. Guard: in scripts/ci, a file that writes stdout never calls process.exit(<expression>)./jobs reads in merge-gating paths paginate and merge (gh api --paginate --slurp … | jq '{total_count: .[0].total_count, jobs: [.[].jobs[]]}'); the two open-PR --limit sites rise and fail loud at saturation. Guard: an unpaginated max-page read of /jobs, unfiltered /check-runs or /check-suites in gate code must compare total_count or carry pagination-exempt; the rest are a shrink-only baseline.ls-files to the other repo-scaled verbs (git diff --name-only, --numstat, git log, git grep -l, the union output) and set the ceiling at the sites it then flags — not opened yet./jobs, /pulls and /check-runs reads paginate.PRs: propflowai #17180 (C), #17182 (B), #17189 (D); local-bin #445 (D, parked until its reviewer is back); agent-smith #827 (D). Pattern A part 2 (widening #17160/#17178's ratchet to the other repo-scaled verbs) waits for #17178 and is not opened yet.