Propflow Propflow Docs
fleetAccepted. The build is in `gera-propflow/local-bin` (the `cleaner` worktree phase, `host-headroom` disk bars, the `fleet-grader` disk row, the `driver-brief` floor line).markdown source

0037 — Worktrees are disposable: the cleaner owns idle and merged ones, and the disk is a measured quantity

What was measured (2026-09-29)

Correction, measured the same evening

.husky/post-checkout already shared one install. From 2026-09-25 until the 2026-09-30 amendment below, every new propflowai worktree got an APFS clonefile of main's node_modules: copy-on-write, blocks shared. Gera approved "share one install" before this was re-checked; it was already true. It has since been replaced by links (see "Linked shared install").

The decision

  1. A worktree is disposable once git has its work elsewhere. The cleaner removes a linked worktree only when all of the following hold:

    • git status --porcelain is empty;
    • HEAD is reachable from origin/<branch>, from origin/main, or from the head of its merged or closed PR (that covers squash merges whose branch was deleted);
    • its PR is merged or closed, or there is no PR and HEAD is in origin/main;
    • no non-dependency file changed in the idle window;
    • no live process has its cwd inside it;
    • no task, loop or lane record names it;
    • it was not created in the last 24h;
    • it is not under a protected root (~/.claude by default: live state and session scratch).

    The edge these guards carry alone. A worktree just branched from main with no commits of its own is clean, pushed and in origin/main, and has no PR, so it passes every test except the 24h birth guard and the idle window. Those two guards are its only protection. Neither may be relaxed (in particular, the birth guard must stay at least as long as the tightest idle window) without first adding a stronger test for the no-commit case, such as requiring a merged or closed PR. Superseded in part by the amendment of 2026-10-03: that stronger test is built (HEAD at or behind its merged/closed PR's head), and for a copy that passes it both guards give way to a 1h quiet window. For every other copy this paragraph stands.

    This is the one place the cleaner deletes rather than moves. The task phase moves because a task record has no other copy; a worktree that passes these tests has one on origin.

  2. Build caches go first, and clones never go.

    • .next, node_modules/.cache and .turbo are reaped from any worktree idle 12h, clean or not. They are pure regenerable output, and they are where the bytes are.
    • A node_modules carrying the hook's provisioning marker is a clone of main's install. It is never reaped: keeping it is free, and deleting it is the stall.
    • An unmarked node_modules is an install of its own, owns its blocks, and is reaped at the full idle window.
    • Two markers, one rule. A linked node_modules (2026-09-30) carries both markers. .pf-shared is written by the link helper, and .worktree-provisioned by the hook once the layout is in place. Either marker means it is never reaped. .pf-shared additionally makes removing the worktree cheap, because it contains only links, so that removal is allowed outside the nightly window. A tree with neither marker is the only one treated as an install of its own.
    • How "marked" is read: the reaper skips a node_modules carrying either marker (.worktree-provisioned or .pf-shared). There is no lock comparison. A marked tree is a hook-provisioned one, either cloned or linked. The 24-of-25 sample, where every marked tree also had main's exact lock, was taken before the link layout existed, and covers clones only.
    • The known edge. A marked clone into which a lane later runs npm install keeps the marker, but by then it owns new blocks. This rule never reaps it, so those bytes are only reclaimed when the worktree itself is removed in the nightly window. Treating a lock that diverges from main's as "unmarked" would close the gap. It is a follow-up, not built.
    • A dirty or unpushed worktree keeps every file of its own.

  3. What git does not have is never deleted, only listed. A worktree that is idle and dirty, or has commits that exist only locally, goes to ~/.claude/jobs/worktrees-needing-human.json. It is Gera's call, never the cleaner's. Reader: the coordinator surfaces the list to Gera. Until a fleet-grader row counts it (the named follow-up), it is a file with one human reader, not an alarm. Superseded in part by 9: an idle held copy is now archived to origin first, after which git does have its work, and it may then be removed. The list keeps everything the cleaner will not act on: copies idle under 24h, copies whose archive failed or is too large, copies on a person's branch, and saved copies still inside their 7 days (each listed with its archive tag).

  4. It runs on the cleaner's existing hourly cadence, with no new launchd job.

    • worker-cleaner --mode=apply (launchd co.propflow.worker-cleaner, hourly) runs the task phase, then the worktree phase.
    • The worktree phase is bounded per run by small caps. The code (WT_MAX_REMOVE, WT_MAX_REAP) is the one place those numbers live.
    • Deletes are paced and stall-probed (local-bin #304). Files are unlinked in batches, and after each batch a git ref write is timed in a private scratch repo. If it is slow, the run stops and writes BLIND why=fs-stall-during-delete. A volume that is already stalled deletes nothing. Git reads run with GIT_OPTIONAL_LOCKS=0, so inspecting hundreds of worktrees never rewrites their indexes.
    • The hourly chained phase is opt-in. It runs only while ~/.claude/jobs/cleaner-worktrees.enabled exists, and that file is created only after a live paced run has shown git ref writes staying fast. Running --worktrees by hand always works.
    • Reaping takes a little every hour on purpose; removal happens only in the nightly window below. On 2026-09-29, deleting about 80 GB in one burst coincided with a fleet-wide stall: git ref writes sat in uninterruptible wait for up to 35 minutes across repos, and all of them released at 22:23:43Z once the bulk deletes and scans stopped. A GC that stalls the fleet is the incident it exists to prevent.
    • A crash in it is logged and never changes the task phase's exit code.
    • Whole-worktree removal runs only in a nightly window (UTC 08–11 by default, 02:00–05:00 in Monterrey). Removal deletes the worktree's clone, which frees almost nothing and carries the stall risk; its benefit is only keeping git worktree list short.

  5. The idle window tightens under pressure. It is 48h normally, and 12h once the volume is at 92% used or more.

  6. BLIND beats wrong.

    • The phase refuses to act, and writes cleaner-worktrees BLIND why=…, when it cannot read either of the two things every verdict needs:
      • the process list;
      • the task records.
    • When the PR list is unreadable it reaps build caches, and unmarked installs past the full idle window, and writes BLIND. None of those depends on PR state. It never reaps a marked (cloned or linked) tree and never removes a worktree.
    • The liveness line cleaner-worktrees mode= is written only when a run actually judged.

  7. The disk is measured and alarmed where the fleet already carries alarms.

    • host-headroom samples disk use every 5 minutes. Past 85% it reports PRESSURE with a disk N%>85% reason, and an unreadable disk is its own reason, never read as fine.
    • fleet-stability already posts host-headroom's reasons verbatim to Slack. No second threshold exists to drift.
    • fleet-grader gains a disk row. It is GAP past 85%, and GAP if the worktree GC has not judged a run in 3h, because scheduled is not judging.
    • driver-brief prints the disk floor as its first line.

  8. Lanes must push before going idle. A worktree whose only copy of a commit is local is protected forever, so the lane that made it should push. The protection is what keeps the rule safe, and pushing is what keeps the disk free. Superseded in part by 9: "protected forever" now means protected until the cleaner has archived it to origin and 7 idle days have passed. Pushing is still the lane's job; 9 is the net for when it doesn't happen.

  9. Amended 2026-09-29: a held copy is saved, not kept forever. Rule 8 depends on a lane remembering to push, and on the mini on 2026-09-29, 161 held copies showed it doesn't. They were mostly helpers that died mid-edit. So the cleaner archives any dirty or unpushed worktree idle past 24h: its whole tree becomes one commit on top of its HEAD, built in a temporary index so the copy is never written, pushed to refs/tags/archive/worktrees/<date>/<name> and read back from origin. A saved copy may be removed only after 7 days idle, in the nightly window, and only while the tag is still on origin and the copy's tree still equals the archived tree. A failed push deletes nothing. A person can approve removals earlier through cleaner-worktrees-queue.json, and the same archive and tree checks still apply (local-bin #308).

    • What an archive holds, and what it does not. It holds tracked edits, untracked files that are not gitignored, and commits that exist only locally. It does not hold gitignored files (.env*, node_modules, build output, local databases), and removing the copy loses them. That is the intended trade: they are regenerable or machine-local by definition. Stashes live in the shared repo, not the worktree, so removal leaves them. So do secrets in ignored files: they are never pushed. A secret in a tracked or non-ignored file would be pushed, and GitHub's push protection then refuses it. The archive fails and nothing is removed.
    • Two clocks, not one. The 24h archive threshold and the 7-day removal threshold are independent of decision 5's 48h/12h window, which still governs dependency reaping and the removal of clean, pushed copies. Disk pressure tightens only decision 5's window. It never shortens the 24h or the 7 days.
    • Retention: archive tags are never pruned automatically. They are small, because they share objects with the repo. Deleting one is a person's decision, taken knowing that a copy removed under it then exists nowhere.
    • The queue: the writer is a person, or a coordinator carrying out a person's recorded approval. The 2026-09-29 sweep wrote it with stranded.py queue after Gera's "Yes, all five steps". The only reader is the cleaner's worktree phase, which applies every check in this decision to each entry. Nothing else reads it.
    • A person's branch (gera/, fede/, jose/) is never archived and never removed by idle age. It is only listed, because their work-in-progress is not the fleet's to publish.

Linked shared install (decision of 2026-09-30)

Why a second decision was needed. Measured on the mini on 2026-09-29, a whole-tree clone of node_modules (200k+ entries) is nearly free on disk. But for its full 70–95 s, git ref writes and new-file creates stall for every process on the volume:

Every new worktree ran one such clone, which at about 275 worktrees a day meant hours of frozen git. Gera chose "Linked shared install" over a warm pool of reusable worktrees. The trade-off as it was put to him:

The record of his reasoning is the choice itself.

The decision: a worktree's node_modules is a link layout into one immutable shared install keyed by the package-lock.json sha256.

  1. Where it lives. The shared install is ~/.cache/pf-node-modules/<sha256>/.
    • It is built once by npm ci (with HUSKY=0, since it is not a git repo), under a per-hash lock.
    • Then it is made read-only (chmod -R a-w).
    • A lock change produces a new hash directory. An existing install is never changed.
  2. The layout. A worktree's node_modules is a real directory:
    • one symlink per package;
    • a real directory of symlinks for each @scope;
    • workspace packages linked to the worktree's own packages/*;
    • .package-lock.json copied, never linked. On 2026-09-29, npm install --dry-run rewrote it through a link into the shared install.
    • .cache, .vite and .vitest as real directories;
    • a .pf-shared marker naming the install.
  3. Turbopack. It refuses modules whose real path is outside turbopack.root. With the root pinned to the project, every page returned 500: "node_modules/next was removed, renamed, or has a broken symlink".
    • next.config.js widens the root to the common ancestor of the project and the shared install, and does so only when node_modules/.pf-shared exists.
    • The main checkout, CI and Vercel have no marker and are unchanged.
    • Verified in a linked worktree: next dev (Turbopack), next build, vitest, tsgo and eslint.
  4. Who provisions. .husky/post-checkout remains the ONE provisioner. It tries the link layout first, through scripts/worktree-node-modules.mjs. It falls back to the old clone path, unchanged, in any of these cases:
    • node or the helper is absent;
    • npm ci fails;
    • PF_NM_MODE=clone is set.
  5. The escape hatch for npm install <pkg> is node scripts/worktree-node-modules.mjs private ..
    • It removes the links, then runs the hook with PF_NM_MODE=clone to make a private copy under the hook's lock.
    • Letting npm convert the tree itself would re-extract every package (measured: "changed 1424 packages").
  6. The cleaner.
    • A linked node_modules is never reaped.
    • A linked worktree may be removed at any hour, because its node_modules is only links.
    • A shared install is kept while any worktree's .pf-shared names it. The one matching origin/main's lock is always kept.
    • An unreferenced install older than 24 h is reaped only in the nightly window, paced and stall-probed.

Measured in a single run (an observation, not a controlled benchmark; the clone path's own fetch time was not separated out). Three worktree creates with the new hook and nothing else running:

Create Time Why
1 70 s the first git fetch (a cold-start cost, not the steady state)
2 5.1 s
3 5.5 s

A ref-write probe sampled once a second (82 samples) and read 0.11 s worst, with none over 1 s. With the clone path, the same operation stalled over 6 s for 70–95 s per create.

Known limit.

Amendment of 2026-10-01: the session that finished cleans up after itself

What happened. The volume filled again: 0 GB free at 22:51 on 2026-09-30, the second time in two days. The cleaner's worktree phase had never been enabled (~/.claude/jobs/cleaner-worktrees.enabled did not exist), so only the hourly cache pass ran. At that point 491 of 688 propflowai worktrees belonged to PRs already merged or closed.

A stopgap built that night without reading this ADR made things worse. It was an hourly sweep plus a parallel day-time rm -rf of whole worktrees, with an override that ignored disk wait under 15 GB free: exactly the bulk delete this ADR forbids. It was retired the next morning. Its two launchd jobs (co.propflow.next-janitor, co.propflow.wt-sweep) were booted out and deleted.

Decision (Gera, 2026-10-01). "Whenever we merge something it should prompt [the session], kind of like the self-compact prompt: it's been merged, go ahead and take down the worktree to clean up, no permission is needed … so I don't get asked."

  1. A merged or closed PR's worktree is taken down by the session that worked in it, minutes after the merge.

    • wt-merged-notify (local-bin, launchd co.propflow.wt-merged-notify, every 120 s) finds the merge and the owning live session.
    • It prompts that session through msg. The prompt carries the standing permission: no permission is needed, and the session must not ask.
    • A worktree with no live owner stays the cleaner's.

  2. The removal goes through wt-done, and wt-done deletes only through this ADR's own machinery.

    • It imports worker-cleaner and deletes with its paced, stall-probed _rm.
    • It removes a worktree outright only when _cheap_to_remove holds: a linked .pf-shared install, or no node_modules.
    • An older full-clone worktree gets only its build output deleted. Build output is the disk that is actually used. The folder waits for the nightly window.
    • On FsStall it stops and says so.
    • So decision 4 (no daytime bulk deletes) still holds.

  3. wt-done refuses rather than guesses, in any of these cases:

    • the caller's cwd is inside the worktree;
    • a session was launched inside it;
    • another process works inside it;
    • its PR is not merged or closed.

    Leftover uncommitted changes are archived locally (~/.claude/jobs/host/wt-archive/<name>-<sha>/), and the branch is never deleted.

  4. This is the fast path, not a replacement.

    • The cleaner remains the backstop for sessions that died, sessions that ignore the prompt, and worktrees with no PR.
    • Enabling its worktree phase (decision 4's opt-in) is still owed.

Amendment of 2026-10-03: a finished copy goes at the next hourly run

Decision (Gera, 2026-10-03). "Remove finished working copies as soon as their PR merges, instead of waiting 48 hours."

What was measured, 2026-10-03 ~09:00. 231 propflowai worktrees. 38 had a MERGED or CLOSED PR, and 34 of those were under 48h old, so decision 5's idle window held them for up to two days after their work was already on origin. The 10-01 fast path (wt-merged-notify → wt-done) covers only a copy with a live owning session; a copy whose session died, or with a stray shell inside, falls back to the cleaner and waited the full 48h. The cleaner's hourly worktree phase (decision 4's opt-in, owed since 10-01) was enabled at ~09:05 that morning after a paced apply run removed 10 with git ref writes staying fast.

  1. A DONE copy is removable at the next hourly run. DONE means three things: its PR is MERGED or CLOSED; its HEAD is that PR's final head or an ancestor of it; and the copy was born before the PR closed (closedAt/mergedAt from the PR list; unknown means never done). The copy holds exactly what the PR carried, and it existed while the PR was open. For a DONE copy the 48h/12h idle window and the 24h birth guard are replaced by a 1h quiet window (WT_DONE_QUIET_H in the code) measured on two clocks: the PR closed at least an hour ago, and nothing other than dependencies changed in the last hour. The edit clock alone was not enough — a copy held quiet for hours and merged at 14:59 would have gone at the 15:00 run, before wt-merged-notify had prompted its owner. wt-merged-notify prompts the owner within minutes of the merge, and the phase runs hourly, so 1h is "the next run" without racing a session that just merged and is still reading its own tree. (local-bin #369)
  2. The stronger test decision 1 required is now built, so its "neither may be relaxed" paragraph is superseded for DONE copies only. HEAD in origin/main alone is not DONE: PRs are matched by branch name, so a fresh no-commit copy on a reused name would have HEAD in main and pass. The ancestry test against the PR's own head excludes that. The ancestry test alone is also not enough, found by the fresh-context review of local-bin #369: a no-commit copy on a reused name created from a stale origin/main (the mini's main checkout has been 200+ commits behind), or after main was merged into the old branch, has a HEAD that is an ancestor of the old PR's head. The birth-before-close test excludes that: a copy born after its PR closed cannot be that PR's working copy. A no-commit copy with no PR keeps both guards; a reused-name copy with commits beyond the PR head fails the ancestry test and keeps the 48h path. Each of the three tests has a fixture that removes the copy when that test is dropped. A copy created and merged within one day goes once merged: the birth guard existed for the no-commit edge, and DONE already excludes that edge, so keeping it would hold exactly the copies merge-day churn creates.
  3. Everything else holds. Clean tree; no live cwd; named by no task, loop or lane record; not under a protected root; PR list readable (BLIND means DONE is never true). Dirty or unpushed copies never take this path, whatever their PR says: they keep decision 9's 24h save and 7-day wait, and a merged-but-dirty copy under 24h idle is still only kept, not listed and not deps-reaped. Pacing, the stall probe, the per-run cap and the nightly window for full (non-linked) removals are unchanged; a linked copy goes any hour, as since 09-30.
  4. The cap stays at 10 per run, deliberately. 38 eligible ÷ 10 = four hourly runs. A linked removal is about 27k entries (measured: 25,128 source files plus 1,538 links); a full one 160k+. The one burst that ever stalled the box was ~80 GB of unpaced node_modules. If merges outpace 10/h at peak (275 creates a day is ~11/h), the follow-up is a separate cap for cheap removals that leaves the nightly full-removal burst at 10, not a blanket raise.
  5. CLOSED is deliberate. Gera's words were "merges"; this amendment covers MERGED and CLOSED because decision 1 has treated a closed PR as "git has its work elsewhere" since 09-29 (the commits sit at refs/pull/N/head and usually origin/<branch>), and the PRs this fleet closes by automation (cleaner --prs, ORPHANED and idle 7d) belong to copies already past the 48h window. What a closed copy loses is its ignored files, the same trade every removal under this ADR makes.
  6. Accepted residual edge, named, not fixed. The only live-use guard is "a process has its cwd inside", and for a DONE copy its window drops from 48h to 1h. A session that works in a merged copy by absolute path, reads without writing for over an hour after the merge, and ignores wt-merged-notify's prompt, loses the copy; its work is on origin. The cleaner does not scan transcripts for owners the way wt-merged-notify does. If that ever bites, the fix is to read wt-merged-notify's owner index here, not to lengthen the quiet window.

Consequences