0040 — Tests run in the cloud; the mini runs only diff-scoped checks, and what finished or died is reaped by its owner
- Status: Accepted. Built 2026-10-03 across four repos (the Build table); amended the same day with the six-layer enforcement design (see the amendment at the end) after a census showed doctrine alone was not holding. Amended 2026-10-10 (Gera 2026-10-10, fleet cleanup step 2): the 300 s
session-cap reap --scheduledclock (#393) is retired, never having been armed;cleaner-jobsnow runs insideworker-cleaner's hourly job instead of its own;branch-updateris deleted. - Date: 2026-10-03
- Deciders: Gera, 2026-10-03 ~09:50: "make sure we really cover all edges of this mini, because the mini is our main driver for our changes and we want it to be a high-throughput system. If the agent flow is leaving jobs and other things like tests that are no longer working, we need ways to correctly garbage-collect and close completed work. We probably shouldn't be running tests all that often, and we could have them run in the cloud for now." The calls below were made by the host-throughput driver under that brief, with the measurements that decided each one, per ADR-0004. The plan was posted in the CI-cost thread (C0C10AS4U15 / 1790993207.945769, 10:06) before any of it shifted load onto CI; no
ci.ymlchange is part of it. - Systems:
pacer,host(session-cap),nudge,cleaner(newcleaner-jobssibling),host-alerts, dotclaudehooks/guard-bash.shrule 16/28, propflowai.husky/pre-push, agent-smith.githooks/pre-push.
What was measured (2026-10-03, 09:40–10:10, 14-core mini, load 28–52, ~45 Claude processes, 34 tmux sessions, 231 propflowai worktrees)
Where the heavy work came from. The pacer's own log for the previous 24 h, releases by class:
| class | runs/day | median s | p90 s | core-hours/day (units × duration) | exit ≠ 0 |
|---|---|---|---|---|---|
| test (single-file vitest) | 2,687 | 5 | 27 | 27.6 | 28.8 % |
typecheck (npm run typecheck, typecheck:scripts, raw tsgo) |
976 | 24 | 64 | 20.2 | 8.3 % |
| pytest (agent-smith) | 87 | 5 | 92 | 7.0 | 48 % |
| lint / other / browser / build | ~200 | — | — | < 1 | — |
About 55 core-hours a day — 2.3 cores continuously — of paced heavy work, on top of what the pacer cannot see. The two busiest sessions were drivers (263 and 239 heavy-minutes each in the day).
Where the filesystem storm came from. A 10 s fs_usage sample at 09:55: 3.28 M filesystem ops, of which 2.30 M (70 %) were python3.13 — the pytest-xdist workers of ONE agent-smith pre-push run under Smith's own claude -p push (/private/tmp/wt-minter-silent: 975 k stat64, 309 k fstat64). git was 312 k (one git grep at 511 % CPU; ~/.gitconfig read 1,228 times in 10 s — hundreds of short git invocations from hooks and ticks), node 125 k, ps 73 k and lsof 23 k (the fleet's own ticks), rm 23 k (worktree deletes). IOPS at the time: 13–14 k tps, 56–104 MB/s.
Where the non-session CPU went. A 120 s cumulative-CPU attribution (every process charged to the claude process it descends from): session trees 4.7 cores; non-session 3.7 cores, of which fseventsd 1.04, the Codex/ChatGPT desktop app 0.70, Apple XProtect 0.33, rm 0.25 (the cleaner's deletes), the local-bin python jobs together 0.23, Finder 0.12, WindowServer 0.12, the agentflow supervisor 0.05, the relay 0.02, Smith's three daemons 0.09. The fleet's own tooling is ~0.4–0.5 core — already under the 1-core target. What costs is not the fleet's jobs; it is the kernel's file-event service, two desktop programs, and the storms above.
fseventsd. A 5 s sample fseventsd at 09:58 put 62 % of its samples in one client thread, client.814 = Finder (fseventsd doing lstat64 + access per event for that one subscriber: 15.9 k lstat64 in 10 s; Finder itself at 22 k getattrlist + 16.5 k fsgetpath per 3 s, resolving event file-ids under /Users/miniclaw). Finder was relaunched without restored windows at 10:06 (NSQuitAlwaysKeepsWindows=false, killall Finder; rollback = open a window). fseventsd stayed at 1.07 cores, but the cost was then spread over ~45 client.<pid> threads, one per Claude session: the kernel's per-client, per-event filtering is the cost, and the lever is event volume (the storms above), not Spotlight — indexing was already off on both volumes, Time Machine has no destination, and there is no per-directory fsevents exclusion on macOS.
Smith's worker. smith-worker.err.log carried 17–128 TMPRL1104 (workflow-task loop starved) warnings per hour through the busy windows of 2026-10-02 20:00–23:00 and 2026-10-03 07:00–09:00; 0–4 per hour at 10:00. The 2026-10-02 amendment of ADR-0034 names the same 28 pytest-xdist workers.
What finished or died and was still here. A tmux session (op-sess-propflow-ef) whose Claude host had exited, sitting at a bare zsh with no children, no client and no lease — invisible to session-cap reap, whose predicate is pane_dead. 12 nudge --_watch watchers reparented to launchd, up to 3.5 days old, at least one for a session that no longer exists. 802 lease files for ~45 live hosts (the lease hook prunes a bounded number per SessionStart, so the backlog never drains); 80 lock files older than a day for sessions with no live host (one from Sep 3). The pacer's prune() returns a dead holder's units but never signals its job, and records pace's own pid rather than the child's, so a killed pace leaves its vitest/tsgo/pytest tree running unaccounted. dock-finisher exiting 1 137 times an hour is not a defect: that is each Driver's Stop hook recounting a board that is short.
The decision
Tests run in the cloud. Locally, a test file may run only if this branch changed it. The dotclaude guard (rule 16) admits a named
vitest run <file>/pytest <file>only when the file is modified, untracked, or committed on the branch relative toorigin/main; an unchanged test is refused with the reason and the CI path. That keeps exactly the two local runs the propflowai runbook already defends — the first run of a fixture you just wrote, and a positive control on a guard you just wrote — and removes re-verification of unchanged tests, which is CI's job and whose verdict already reaches the session by push (ADR-0039).npm run test:affectedremains the sanctioned diff-scoped selection. The guard also learnsxargs(andfind -exec) as wrappers, in rule 16 and in the pacer prefilter: a 27-minute, 2-core… | xargs -0 npx vitest run <25 files>ran unjudged and unpaced under a driver today. The guard fails open, logged, whenorigin/maincannot be resolved — a guard that wedges a session is worse than one that lets a test run (ADR-0034 §5).Type-check stays local, once, right before push. The brief listed
tsgowith the suites. It stays because of one measured fact: the bot review (claude-code-review.yml) fires on everysynchronizeafter a 90 s settle and does not gate on a green Type Check. At 976 runs/day and 8.3 % red, moving it would add ~80 red pushes a day, each a paid bot review of code the compiler would have refused in 24 s. Gera's own carve-out (2026-09-02, recorded in the guard: "we can maybe compile, but running tests just adds more friction") says the same. The pacer's memory budget (typecheck = 8 GB against 20) is what makes it safe to keep; the doctrine says once before push, not per edit. This is the one point where the brief was adjusted by a measurement, and it is a CI-cost question in Fede's lane if the review ever gains a typecheck gate — then this clause flips.A pre-push hook runs nothing heavy. propflowai's
.husky/pre-pushdrops its warn-only local mutation-control run (check-mutation-control.tsexecutes vitest per control touched; 218 solo reruns on record, nearly all load flakes) — CI'schecksstep is the one blocking copy. agent-smith's.githooks/pre-pushkeeps ruff + mypy (the compile class) and stops running the affected pytest set; the one thing CI cannot run — the 67 cross-repo tests that need a propflowai checkout — still runs locally when affected, paced. The structural-change fallback no longer runs the full suite locally for any reason.What finished or died is reaped by the thing that already owns it, never by a new reaper.
- The pacer records the child pid and, each tick, TERM→KILLs a holder's tree when its session has no live host (the fleet's one liveness predicate — pid + start time — never a new one; a lease file existing proves nothing), or when the job has run past a per-class maximum and its tree's CPU time has not advanced for ten ticks (the hung shape agent-smith's hook header records: 99 % reported, every worker at 0 %). A live session's job that is still burning CPU is never signalled, whatever its runtime — a 27-minute vitest run is foreground work. Every reap is logged, written to
reaped.jsonl, announced through the fleet's one poster, counted instatus.json, and areap-judgedline proves the reaper looked even when it found nothing (ADR-0009). session-cap reapgains a second fact besidepane_dead: every pane is a bare login shell with zero children, no client attached, and the pane's session id has no live host; idle time only dampens (≥ 30 min, so a relaunch is never raced), and fleet-medic's resurrection path is checked first. "Looks idle" stays rejected, as its docstring says (2026-08-08: seven of ten live loops retired on a guess).nudge --_watchexits by itself when its session has no live host on two consecutive checks. Nothing kills a watcher from outside.cleaner-jobs(a sibling of the cleaner with its own hourly launchd job, so it does not collide with the worktree work being done onworker-cleanerthe same day) removes, from an allowlist only: leases of dead hosts older than 7 days, anddecision-wake/nudgelocks of dead hosts older than 24 h. Capped per run, every removal logged, a run line written only when it judged,BLINDwhen it could not read.
- The pacer records the child pid and, each tick, TERM→KILLs a holder's tree when its session has no live host (the fleet's one liveness predicate — pid + start time — never a new one; a lease file existing proves nothing), or when the job has run past a per-class maximum and its tree's CPU time has not advanced for ten ticks (the hung shape agent-smith's hook header records: 99 % reported, every worker at 0 %). A live session's job that is still burning CPU is never signalled, whatever its runtime — a 27-minute vitest run is foreground work. Every reap is logged, written to
The host says what it spends on non-session work, and names the culprit when it is too much. The pacer's tick publishes
cores: {session, nonsession, tooling, top_nonsession}from cumulative CPU-time deltas (onepsper tick, the one it already reads;ps %cpuis a lifetime average and lies for a 5-day-old daemon).host-alertspoststooling-over-budgetwhen the fleet's own tooling passes 1 core for 10 minutes, naming the top three commands, andfseventsd-hotwhen fseventsd passes 0.75 core for 10 minutes, naming the hottestclient.<pid>from a boundedsample fseventsd— the only way the attribution above was found. Signals, not calls (ADR-0003).
What was deliberately not done
- No cap on sessions, and no reaping of live foreground work. Gera's standing rule, unchanged from ADR-0034 and memory-guard.
- No Spotlight/fsevents exclusion. Spotlight indexing was already off; fsevents has no per-directory exclusion; the per-client cost is a property of how many Claude sessions subscribe (one each) times how many events the box generates. The decision attacks the events (the pytest storms, the pre-push suites) and makes the client visible, rather than claiming a knob that does not exist.
- Nothing in
ci.yml. One swarm's ci.yml PR in the line at a time, owned by session 002; this ADR's load shift was announced in that thread first. - No change to worker-cleaner or ADR-0037's worktree rules. Another lane owns those today.
- The Codex desktop app (0.7 core idle) and Apple XProtect (0.33) were measured, not touched. A reflex may not kill either; the
tooling-over-budgetpost shows the top non-session consumers so a human sees them.
Build
| Lane | Repo | PR | State |
|---|---|---|---|
| A — pre-push drops mutation controls; runbook + CLAUDE.md rule | propflowai | PropFlow-Technologies/propflowai#16046 (hook), #16047 (doctrine; rewritten before merge to say the changed-file rule is ENFORCED, not advised) | in the merge line at the time of this amendment |
| B — guard rule 16 changed-file predicate + xargs; worker/local-review doctrine | dotclaude, local-bin | gera-propflow/dotclaude#161 (guard) + #162 (local-review); gera-propflow/local-bin#373 (worker skill — a symlink into local-bin) + #375 (pace classifier learns xargs / find -exec: that, not the guard's prefilter, was the unpaced hole) |
merged |
| C — pre-push: ruff + mypy + cross-repo only; CLAUDE.md gate | agent-smith | PropFlow-Technologies/agent-smith#749 (hook + affected_tests.py --only-from), #748 (gate), #754 (17 truncated parametrized ids in ci/cross-repo-tests.txt rewritten — as positional ids they matched nothing and xdist emptied the run silently) |
#748, #749 merged; #754 in line |
| D — pacer reaps dead-owner and hung jobs | local-bin | gera-propflow/local-bin#371 + #376 (dead owner, judged by lease + liveness); #379 (hung job); #381 (unreadable lease = unknown; no CPU reading = no judgement) | merged, live: REAP-JUDGED every tick, 0 reaped on the first hour — catalog + ADR-0034 amendment 2026-10-03 |
| E — session-cap shell-only reap; nudge self-exit; cleaner-jobs | local-bin | gera-propflow/local-bin#382 (fleet-medic host-alive, the one three-valued liveness answer the three reapers share), #372 (session-cap: a host that exited and left its shell is a fact), #377 (nudge --_watch exits when its host is gone — applies to watchers armed after it; the long-lived ones keep their old script until re-armed), #378 (cleaner-jobs, locks only — leases stay, because the medic resurrects through them for 12 h), #393 (a 300 s session-cap reap clock, watch-only until ~/.claude/jobs/host/session-cap-reap.armed exists; arming is decision b91047893 in front of Gera); catalog agentflow #274, #280 |
merged, live: REAP removed op-sess-propflow-ef — host exited (sid 3de5e881), shell idle 3569m; cleaner-jobs first run judged 99, removed 61 |
F — cores split in status.json; tooling-over-budget + fseventsd-hot |
local-bin | gera-propflow/local-bin#374 (cores), #380 (rules), #383; PropFlow-Technologies/agentflow-relay#273 (catalog) | merged, live: first reading session 5.03 · nonsession 3.29 · tooling 0.30; fseventsd-hot posted at 11:22 naming Finder at 4 % — the load had spread across the sessions' own client threads |
Amendment 2026-10-03 — "tests don't run on the mini" is enforced in layers, not believed
- Deciders: Gera, 2026-10-03 ~11:45, after the morning's doctrine landed and a census still showed sessions running tests locally (at 11:30: six vitest processes under a propflowai pre-push, one 25-file sweep at two cores for 1 h 45 min under a live driver — stopped by hand at Gera's OK; the "six orphans" in the same reading were a false count from
pgrep -fmatching driver prompts that mention vitest): make it bulletproof — doctrine alone isn't enough; enforce it in layers so no single gap reopens it. Recorded by the host-throughput driver under that brief. - What the first two minutes of the test census showed (
co.propflow.test-census, the coordinator's 4-day sampler, self-removing 2026-10-07): the same thing by three doors — a driver's pre-push running a drift test (via pre-push), a driver'snpm run test:affected(via pace), and a session's own Bash running one test file (via session-bash). Three entry paths means three layers, and a proof that reads all of them.
The layers
- The Claude guard denies every heavy check (dotclaude
guard-bash.sh, beside rules 16 and 28). In any session's or sub-agent's Bash call, through every wrapper the rule already knows (env,timeout,nice,pace,npx,npm exec,pnpm,bunx,uv run,xargs,find -exec,bash -c, ansshpayload): vitest / jest / pytest sweeps;npm|pnpm|yarn|bun test*andnpm run test*— includingtest:affected, which is retired as a local path;tsc/tsgo/vue-tscover the project andnpm run typecheck*;next build,npm run build*,playwright test,remotion render. Decision 2 of this ADR is flipped by this call: type-check moves to CI with everything else. The cost it carried (the bot review fires onsynchronizewithout a type-check gate, so ~80 of today's 976 daily type-checks would have become red pushes with a paid review each) is accepted and stated; it is Fede's CI-cost lane's to reduce by gating the review on a green Type Check. Still allowed:tsgo --version,vitest list,--collect-only, eslint on named paths, the diff-scopedtsx scripts/check-*.tsthe pre-push runs (seconds). One exception, one config. Up to N explicitly named test files that are changed in this branch (modified, untracked, or committed relative toorigin/main— the predicate dotclaude#161 introduced), still routed through pace. N islocal_tests.max_named_changed_filesin~/.local/bin/pace.json, read by the guard and by the pacer alike; default 1, because the first run of a file you just wrote is authoring feedback and two is a sweep; 0 means cloud-only with zero exceptions, on the same code path (the loop admits nothing).CI_PUSH_GUARD=1stays exempt (Fede's first-push guard set, agreed in the CI thread at 10:07). The deny text tells the agent exactly what to do: push, and CI's verdict reaches the session by push (ADR-0039); orcloud-test <files…>(layer 5). Every deny and every admitted exception is a line in~/.claude/jobs/guard-bash-local-tests.log; fail-opens stay inguard-bash-failopen.logand in the hook's own note. - The mini's pre-push is independent of the branch. Measured 11:35: 186 of 207 propflowai worktrees and 13 of 15 agent-smith worktrees (and all of Smith's own
/private/tmp/wt-*lanes with a checkout) still carried the heavy hook their branch was cut with, and would until each rebased;branch-updaterskips propflowai on purpose (strict=false). So local-bin gains a machine-level hooks directory per repo, pinned bycore.hooksPathin each repo's common git dir — one write covers every worktree — and re-asserted every 300 s by ahooks-pinlaunchd tick, because husky'spreparere-sets the path on everynpm install. The machine hook execs the branch's own hook only when it carries# pre-push-profile: cheap-2026-10-03; otherwise it runs the canonical cheap set from the worktree's own scripts (propflowai: the five diff-scoped checks, the behind-count and the push pacer, never the mutation controls; agent-smith: ruff, mypy, and the touched cross-repo tests only) and says so in one line. agentflow-relay and local-bin have no hooks (verified). - The pacer refuses what the guard never saw. A request of class test / pytest / typecheck / build / browser / render — from a script, a cron, a hook, Codex, anything — is refused unless it is the same ≤N-changed-files exception, with the same deny text, a
refusedlog line andrefused_24hinstatus.json. This layer is the backstop, so it fails closed: a request it cannot check (no git, noorigin/main) is refused and the reason logged — the opposite of the guard, which fails open so a session is never wedged (ADR-0034 §5).watchmode logswould-refuseand runs;DISABLEDruns. A second exception, bounded by a file and not by N: the machine hook's cross-repo pytest set (layer 2) —pace --fg -- uv run --frozen --extra dev pytest -q <ids>with every id listed inci/cross-repo-tests.txtatorigin/mainof that repo, so a branch cannot widen it — is admitted (admittedlog line,admitted_24hin status), because it is the one set CI cannot run (local-bin part 4b). - The reaper stops what is orphaned or too old — in the pacer's tick, never a new daemon (decision 4 of this ADR). Beyond the dead-owner and hung rules already live (#371/#376/#379/#381): an unpaced test or type-check tree (started before the classifier knew its wrapper, inside a hook, or by a pace that was SIGKILLed with its session) is judged by walking its ancestry to a
claudeprocess and asking the fleet's one liveness question of that session; no ancestor or a dead one is an orphan and is TERM→KILLed with the same evidence row and Slack line. And any such tree older thanmax_run_s= 1200 s (20 min), held or unpaced, is stopped whatever its CPU — Gera's call, overriding the never-kill-foreground rule for this class only; a session that still wants the answer hascloud-test. Unknown ancestry keeps the tree and is counted. cloud-test <files…>is the escape hatch, so no session has a reason left to run locally: aworkflow_dispatchworkflow in propflowai (and agent-smith) that runs only the named files on the session's branch on the cheap runner with a 10-minute cap, and a local-bin CLI that dispatches it, waits at 60 s intervals, prints the verdict and the failing names, and logs each call to~/.claude/jobs/cloud-test/calls.jsonl. It is CI-facing, so it was proposed in the CI-cost thread (C0C10AS4U15 / 1790993207.945769, 11:45) and its PRs stay unarmed until that thread says go; nothing inci.yml.- The proof is a signal, not a belief. The pacer's tick publishes
local_tests.cpu_s_last_hour {exempt, nonexempt}from the per-tick CPU-time deltas it already computes, classifying processes the way the test census does (reused, not re-derived — the census is catalogued as temporary and this supersedes it), where exempt means descended from a pace holder admitted as the named-changed-files exception or carryingCI_PUSH_GUARD=1.host-alertspostslocal-tests-ranwhen non-exempt local test CPU passes about one CPU-minute in an hour, naming the offender, its session, and the layer that should have caught it, and clears when an hour reads zero. "Bulletproof" is the day that line reads zero.
What no layer closes, said plainly
- A human typing in a raw terminal on the mini: layers 2, 3, 4 and 6 apply; layer 1 does not.
- The Codex desktop app and
codex execruns: not under the guard; the pacer sees them only when they call a heavy command by name; the reaper and the signal see them regardless. - A test runner invoked under a name none of the classifiers know: layer 6 reports it as non-exempt CPU by
comm, which is the cue to teach the classifier. - Fede's Mac: no
pace, and the guard fails open there by design; the layers protect the mini. - A pre-push hook that a branch rewrites to bypass the marker convention: layer 3 refuses what it spawns, and layer 6 reports it.
Build (this amendment)
| Layer | Repo | PR | State |
|---|---|---|---|
| 1 — guard denies all heavy checks; N config | dotclaude, local-bin | gera-propflow/dotclaude#165 (9ff4cd9: rule 16 refuses vitest/jest/pytest sweeps, test* scripts incl. test:affected, tsc/tsgo/typecheck*, builds, playwright, remotion, through every wrapper; one exception, N from pace.json; 917-case matrix; a Codex read found seven bypasses, all closed), #167 (b690a07: rule 29 carried upstream verbatim — see Lessons), #168 (ca64321: the check-mutation-control.ts runner shape — see Measured after; runtime 2df1c9d); local-bin#387 (6138a56: worker/architect doctrine, the local_tests.max_named_changed_files: 1 key) |
live on the mini since 17:36Z (runtime 92a464a); first live deny deny typecheck 1 at 17:36:26Z; verified from the driver: typecheck, an unchanged named test and test:affected denied, a diff-scoped tsx scripts/check-*.ts allowed |
| 1b — the exception has a per-session rolling budget | dotclaude, local-bin | dotclaude#169 (1629f76: 3 named-changed runs per 900 s per session, from the ledger the rule already writes), local-bin#397 (6c5c82b: the same budget at the pacer, counted from its own admissions) and #398 (75a43f8: pace.json carried TWO local_tests blocks and JSON keeps the last — merged into one) |
live 18:1xZ; proven with four calls (the fourth denied); first real fourth-run refusal at the pacer 14:24 (budget 3/3 window=900s). Found by the census after layer 1 went live: one session ran 16 single-file runs in 30 min, each "one changed file" |
2 — machine-level hooks + hooks-pin |
local-bin, propflowai, agent-smith | local-bin #384 (githooks/ + hooks-pin + launchd), agent-smith #755 (marker), propflowai marker after #16046 |
catalogued: hooks-pin |
| 3 — pace refuses non-exempt heavy classes | local-bin | gera-propflow/local-bin#389 (b89d4dd: refuse test/pytest/typecheck/build/browser/render unless ≤N named CHANGED test files; fails closed on unknown), #392 (11c3da3: the cross-repo pytest set is the second exception, admitted only in the hook's exact form and only for ids in ci/cross-repo-tests.txt at origin/main — a branch cannot widen it), #395 (16dedd7: scripts/check-mutation-control.ts is class test; a named exception must look like a test file) |
deployed; first live refusal 12:01:41 (npm run typecheck, a live driver); 27 refused / 3 admitted by 12:43, one refusal in a personal repo — host-wide by design |
| 4 — unpaced orphans + 20-minute hard max | local-bin | gera-propflow/local-bin#385 (fb060e6; reap.hard_max_classes in pace.json, rollback = empty the list), on top of #371/#376/#379/#381 |
deployed; REAP-JUDGED every tick; 0 reaped through 12:43 — every holder alive by its lease, nothing orphaned, hung or past 20 min |
5 — cloud-test workflow + CLI |
propflowai, agent-smith, local-bin | propflowai#16089 (0dbe0f4c — workflow + a contract test pinning the per-ref concurrency group and the #15881 cheap-runner expression, with two mutation controls; it earned its review the normal way once it touched an ordinary file, after the CI-line owner declined a named-exception merge — an override of a failing required check is not a sound path), agent-smith#759 (14322833; #756 parked as the reason= follow-up), local-bin#394 (77c585b dispatch + ledger), #391 (a60948f --wait, merged by hand on its head after three 🟡 non-blocking rounds — the three-rounds rule, and local-bin's lander is broken), #399 (1dccf7c: a repo whose default branch has no workflow gets a named refusal and "push"), #400 (ee44829); catalog agentflow#279 (3720c96) |
live: one real call from the mini ran 7 propflowai tests in the cloud in 136 s wall (run 37149321880, CLOUD_TEST_RESULT state=success passed=7 failed=0), 24 for agent-smith (run 37147259923) |
6 — local_tests signal + local-tests-ran alert |
local-bin | gera-propflow/local-bin#386 (07a2279: status.json.local_tests, exemption stamped by pace's own admission, never a caller flag), #388 (9686900: local-tests-ran at > 60 CPU-s/hour non-exempt); catalog agentflow#277 |
live: first hour exempt 0.4 s / non-exempt 0 s; at 14:53 exempt 4 s / non-exempt 18 s (under the alert bar) |
| 7 — the runner itself refuses an unadmitted run | propflowai, agent-smith | propflowai#16270 (scripts/lib/runner-admission.ts + vitest.config.ts + check-ddb-escapes.ts), agent-smith#771 (tests/_runner_admission.py + conftest.pytest_configure); catalog agentflow#291 (this row + _runner_refusal_note under the pacer in SYSTEMS.md) |
proven live 2026-10-04 01:35Z: a codex exec --dangerously-bypass-approvals-and-sandbox shell running one vitest file was refused (exit 1, one line in ~/.claude/jobs/runner-refusals.log); the same file through the guard's exception (pace admitted, PACE_ADMITTED=changed-files) ran 13/13 |
Layer 7, why it exists after six. Measured 2026-10-04 00:30–01:20Z with all six layers live: local_tests.cpu_s_last_hour.nonexempt still read 61–78 s. The runs came from scripts/check-ddb-escapes.ts (one npx vitest run <file> per touched file) spawned inside codex exec --dangerously-bypass-approvals-and-sandbox shells — 22 such runs from one session in the window — which no PreToolUse hook sees, and from vitest worker trees outliving pace's release. Layers 1 and 3 judge a command they are handed; layer 7 judges the process from inside it: each repo's test-runner config refuses, before collection and with exit 1, when CI is unset, ~/.local/bin/pace.json (the pacer's own local_tests.refuse_classes — one policy, now read by the guard, the pacer and both runners) names its class, and the process carries neither pace's PACE_ADMITTED stamp (lane D; pace drops an inherited copy, so a caller cannot self-admit through it) nor CI_PUSH_GUARD=1. It fails open by absence — the policy file exists only on the mini, so Fede's Mac and CI run as before — and fails closed on a present-but-unreadable file, logging every refusal to ~/.claude/jobs/runner-refusals.log so layer 6 can count what the guard never saw. What it still does not close: a raw terminal forging the stamp by hand, and any runner neither config loads.
Measured after (2026-10-03 12:37, load 50, 14 cores — the same instruments as the morning)
| 09:52 (before) | 12:37 (layers 1–4 live) | |
|---|---|---|
| session trees | 4.7 cores | 3.3 cores |
| non-session | 3.7 cores | 5.4 cores — of which next-server 1.75 (a session's dev server, under the one-hour reaper), fseventsd 1.13, the Codex app 0.61 |
| fleet tooling | ~0.4–0.5 core | 0.72 by the pacer's own split (it counts everything tmux hosts outside a session tree, so dev-server shells land here) |
| fs ops / 10 s | 3.28 M (python3.13 70 %) | 2.02 M (git 35 %, Python 26 %, node 18 %) — the pytest storm is gone |
Smith TMPRL1104 per hour |
17 (07h), 39 (08h), 10 (10h) | 0 (11h), 0 (12h) |
| pacer counters since 12:01 | — | refused 27, admitted (cross-repo) 3, reaped 0 |
The after-reading is not a clean win and this ADR says so: load stayed 40–50 because sessions, dev servers and fseventsd still cost what they cost; what fell is the thing the decision targeted — local test and pytest CPU, the filesystem storm it carried, and Smith's starvation, which read zero for two hours.
What the census caught after layer 1 went live, and what it taught. Between 12:36 and 12:39, seven vitest runs started from one driver's own Bash with no guard line: timeout 900 npx tsx scripts/check-mutation-control.ts --base origin/main. That script is a test runner wearing the name of a diff-scoped check, and layer 1's allowlist ("tsx scripts/check-*.ts is seconds") admitted it; pace never saw it either. Closed on both sides the same hour: dotclaude#168 refuses every mutation-control shape, local-bin#395 classifies the script as test. The 14 "via pre-push" runs at 12:03–12:11 were one git push that had begun before the 12:01 hooks pin and ran its branch's old hook to the end — expected, not a gap.
Lessons recorded
A mini-only commit on a guard file blocks every future guard update.
update-dotclaude.pyrefuses a mergedhooks/guard-bash.shthat is not byte-identical to one side (runtime HEAD or main). The runtime carried rule 29 locally (Gera's mini-only merge-queue cap), so #165 — and any guard change after it — produced a merged blob matching neither side and was refused. The fix was to carry rule 29 upstream verbatim (dotclaude#167); the rule is: guard content lives in main or it does not live on the mini.An allowlist named by file pattern admits whatever wears the pattern.
check-*.tswas a class of cheap checks until one of them ran vitest. Layer 6 is what makes such a miss visible; it is not optional.Measure by three doors, not one. The census's
entryfield (pre-push / pace / session-bash / script / orphan) is what turned "sessions still run tests" into three specific layers.An exception judged per call is a sweep judged per minute. "≤1 changed test file" admitted 16 runs in 30 minutes from one session. The budget is per session and per window, and it is enforced at both layers that admit the exception, each from its own ledger.
What the
CI_PUSH_GUARD=1carve-out covers: named changed test files only — never a type-check; the guard deniesCI_PUSH_GUARD=1 npm run typecheck, and the doc says so.The budget keys on the first 8 hex characters of
session_id. A call the hook cannot attribute is admitted and logged with a blank id; real session ids are hex UUIDs, synthetic probes may not be.A duplicated JSON key is a silent override. Two
local_testsblocks in pace.json meant python and jq read only the last; an edit to the first would have changed nothing, for the guard and the pacer alike.A workflow-file-only PR cannot earn the required
reviewwhileclaude.ymlis disabled, and an admin override is not a sound path. The cure that holds is to give the PR an ordinary file the reviewer reads — here, the contract test the workflow deserved anyway. The standing hold is Smith's to cure, with an attended merge from Fede's side.gera-propflow/local-bin and dotclaude have no reviewer credential (
CLAUDE_CODE_OAUTH_TOKENis rotated for PropFlow-Technologies only) and local-bin'sland-on-greenrunsgh pr mergewith no checkout, so every fleet-tooling PR today merged by hand after a Codex read recorded in a PR comment. Two infra items, named, not fixed here.
Measured at close (14:53, all six layers live on both repos)
Load 20 (morning 28–52, peak 105 at 10:19). Session trees 0.6 cores; non-session 3.7 (fseventsd 1.05, the Codex app, one dev server under the hour reaper); fleet tooling 0.7. Guard ledger for the day: 1,008 refusals — 456 suites, 305 type-checks, 78 builds, 27 mutation-control, 22 e2e, 22 over-budget — against 206 admitted single changed files. Pacer: 35 refused, 68 admitted, 0 reaped (nothing qualified all day). Census starts per hour: 21 (11h, before the guard) → 66 (12h, 43 of them session-bash before 12:36) → 11 → 7. Smith TMPRL1104 per hour: 17, 39 (07h, 08h) → 10 → 5 → 1 → 0. Still in the merge line at close: propflowai #16122 (the written rule brought in line with the guard) and #16090 (the pre-push marker). Decision in front of Gera: b91047893 (arm the five-minute dead-terminal cleanup, watching and logging now).
Row 4, reopened 18:00 (fseventsd was growing, not just hot)
The morning's row 4 measured fseventsd at ~1 core and attributed it to client
churn (one Finder client at 62 % of samples; after killall Finder, ~45
claude-session clients). At ~18:00 CDT the coordinator reopened it on new
evidence: fseventsd (pid 100, up 5 days) had grown to 15 GB RSS and was the
likely trigger of memory-guard's 17:57 HOLD. Gera approved a restart by hand:
killall fseventsd (SIGTERM) was ignored — measured — and kill -KILL
worked; launchd respawned it (pid 1185, 10 MB, 0.3 % CPU) and the HOLD cleared.
Cause, as measured. Client churn, not any one client. 8 h of log show:
5,854 fsevent_add_client: no bundle id available for pid N lines from 781
distinct pids, 3 removals logged, ~100 new client pids an hour; 772 of the
adders were already dead when counted (596 pids with 2–5 adds, 176 with more,
max 37), while the live sessions carry only 5–9 adds over hours. Pid-based
attribution over an hour-old log is unreliable (a 16 s sleep inherited 31
adds by pid reuse), so the live attribution was taken once by hand: a 120 s
log stream beside a 1 s ps, each add joined to the command seen within 2 s
— 27 adds from 4 pids, claude -p (Smith's runs) 17 (63 %), the Codex CLI
(codex exec) 10 (37 %); ≈ 810 adds/h, the same order as the log. Short-lived
processes open an FSEvents stream and exit; the daemon keeps the client state
and never gives it back.
Two facts measured at 19:0x set the bars. (1) Growth after the respawn is episodic, not steady: 9.87 → 10.50 MB in the first 10 min (≈ 3.6 MB/h; a steady climb to 15 GB in 5 days would be ≈ 125 MB/h) — it grows in bursts, a storm or a client that stops draining. (2) fseventsd's ~1 core all day was its own accumulated state; a KILL + launchd respawn took it to 0.003 cores and 10 MB. The fresh daemon (pid 1185, 11 min old) measured 0.003 cores over 30 s against 1.04–1.13 cores all day, and that cost never tracked fleet load — 09:52 sessions 4.7 cores / fseventsd 1.04; 14:53 sessions 0.63 / fseventsd 1.05 — it tracked the daemon's age and accumulated client state. So the restart is the row-4 lever, for the morning's CPU finding as much as for memory, not a safety valve.
What shipped (the same arm-file pattern as the mds restart, ADR-0034):
host-alerts fseventsd-rss (local-bin#407) — RSS ≥ 2 GB for 10 min posts once
with RSS, age, CPU and the top adders from a bounded live trace run only on the
posting tick, recovering under 1 GB; memory-guard restarts fseventsd
(local-bin#406) at RSS ≥ 4 GB, or age ≥ 12 h and ≥ 0.75 core sustained over
30 min and ≥ 1 GB (the CPU form, without restarting during a legitimate short
storm), never more than once per 6 h (a restart loop would be worse than the
bloat) — SIGKILL (TERM is ignored), a wait for launchd's respawn, the new pid
verified, RESTART fseventsd old=… rss=… cpu=… age=… new=… newrss=… newcpu=…
logged with the before and after so the next reader can check this claim, and
posted; armed at deploy on Gera's approval of the hand restart, rollback =
delete ~/.claude/jobs/host/memory-guard.fseventsd-restart.armed. What this
does not do: reduce the churn — that is the sessions' and Smith's stream usage,
a separate decision.
Consequences
- Sessions push sooner and more often; the CI line sees more pushes from sessions that used to re-verify locally. The push pacer flattens bursts. Expected, announced, and the reason the thread was told first.
- A session that changes a test file still gets its authoring feedback in seconds; a session that wants to know whether an unchanged test still passes waits for the PR's checks and is woken when they land.
- Heavy work on the mini should fall by roughly the test and pytest rows above (about 35 core-hours a day) plus the pre-push suites the pacer never saw; the after-measurement is the driver's final report, taken at comparable load.
- A job whose session died no longer runs to completion for nobody; a shell a host left behind no longer holds a tmux session; the jobs directory stops growing by the dead.