Propflow Propflow Docs
fleetAccepted — shipped in **watch** mode 2026-09-28; enforce is switchedmarkdown source

0034 — Heavy work takes turns by fair share, and the guard rewrites the command

The problem

Measured 2026-09-28 ~00:05 on the mini (14 cores: 10P + 4E): load ~30 sustained, spiking to 60. At one moment 24 vitest processes, 17 render processes (Remotion/ffmpeg/headless chrome), 5 next dev servers, tsgo type-checks at 7–10 GB each, eslint and Playwright — each started by a different session with no knowledge of the others. When bursts lined up, 20–34 processes sat in uninterruptible disk wait, background jobs starved (the ccswitch snapshot went stale and alerted), and the kernel panicked twice that week from userspace-watchdog stalls.

Per-tool caps already existed and did not help: propflowai's vitest.config.ts caps one run at half the cores and narrows under load, but N sessions each take their own "half". Nothing had a view across sessions.

Gera's constraint, the same one memory-guard was built under: no cap on the number of agents. Only heavy work takes turns.

Decision

  1. The unit of fairness is the session. Accounting is per CLAUDE_CODE_SESSION_ID; a session's subagent shells inherit the parent's id, so a session and its subagents are one account. Only sessions with heavy work running or waiting now count. A session alone may use the whole pool (borrowing); when units free, they go to the waiting session holding the fewest — furthest below its equal share — with ties first come, first served. No preemption: borrowed units come back when the job ends. The order is strict (a small ask does not jump the head of the line), so big asks are never starved.

  2. Units, not slots, and the balloon is real. Each class has a cost; a grant may be smaller than asked (down to the class minimum) and the child's parallelism is set from the grant — VITEST_MAX_FORKS/VITEST_MAX_THREADS (read natively by vitest 3.2.7 and by propflowai's own config), GOMAXPROCS for tsgo — plus nice -n 5. Long jobs (render, build) share a lane capped at min(4, capacity/2), so they can never fill the pool.

  3. Capacity follows the machine, not a constant. Base = cores, floor = cores/2; multiplicative decrease while the uninterruptible-wait count or load/core is at its high mark and pacer holds work (shrinking an idle pool cannot reduce load it did not cause); additive increase once neither has been high for a minute. A band, not a low mark: the review found that a separate low mark never recovered at this box's baseline (sessions alone sit at load/core 2–4), so one spike would have halved the pool for the day. memory-guard's HOLD file stops all new grants.

  4. Enforcement is transparent: the guard rewrites the command. dotclaude guard-bash.sh (rule 28) asks pace hook and returns the rewritten command as PreToolUse updatedInput (pace --class X [--fg] -- <original>). Verified on Claude Code 2.1.283 by a headless run: the rewritten command is what runs, with or without a permissionDecision; none is sent. (The permission matcher then sees the rewritten string; that is moot while the fleet runs bypassPermissions, and would matter only to an allow/deny rule written against npm …/npx ….) The guard asks pace only for calls it is not denying, and cuts it off at 3 s. Sessions do not need to know pacer exists.

  5. It always fails open, and never blocks unbounded. No pace on PATH (the Mac), a broken pace, a broken config, mode: off, the kill switch file, a state lock busy past its bound (1 s watch, 10 s enforce), or the max wait → the command runs exactly as typed, and the log says so. Watch mode starts the child before it touches any shared state. A foreground Bash call waits at most 75 s and the hook extends that call's timeout to cover it, so queueing never silently eats the 2-minute Bash limit; a busy queue tells the session to use run_in_background.

  6. No daemon in the decision path. State is one JSON file under one flock; whoever holds the lock hands out every grant that fits, using the same pure function pace replay uses, and waiters poll about once a second to notice. Every entry is stamped with its pid and a flock its process holds, so a killed holder's units return on the next lock and a reused pid cannot keep them. The launchd tick exists only for readers (status.json freshness, pressure samples).

  7. status.json is a published contract (fields in SYSTEMS.md), consumed by the Systems page and by host-alerts, which posts when the oldest waiter passes 10 minutes. Consumers integrate by reading it — signals, not calls (ADR-0003).

Rollout

Shipped in watch mode: every heavy command is wrapped, runs immediately and unchanged, and its demand is logged (request / grant with would_wait / release). pace replay re-runs the scheduler over that log at chosen pool sizes and reports p95 wait per session, peak concurrency by class, and grants that borrowed versus stayed within share. Enforce is one config value ("mode": "enforce" in ~/.local/bin/pace.json); the kill switch is touch ~/.claude/jobs/pacer/DISABLED.

Review

An independent review (Fable, 2026-09-28) confirmed the scheduler's properties and the guard's fail-open paths, and found three defects fixed before merge: an unbounded lock wait (watch mode could wait), AIMD that never recovered at this box's baseline, and nohup being moved inside pace where a SIGHUP during the wait would kill it. Known limits it recorded: a grant is sized at its instant and does not re-inflate when the pool empties; pace replay uses durations measured without coordination and a fixed capacity.

Alternatives rejected

Consequences

Amendment 2026-10-02 — the night Smith's worker starved twice

What happened. 21:33–21:41 CDT Smith's Temporal worker logged [TMPRL1104] Workflow task exceeded 5 seconds (verbatim from smith-worker.err.log; the figure in that warning is the SDK's per-task deadline, and the agent-smith#736 alarm quotes it at 10 s — the line, not the number, is the receipt), ~20 "Failed handling activation", and a finished reply was discarded; 22:47 a 20 s activity waited 21 s to start. Measured: the 21:30 episode was a swap storm (memory-guard: swap 10.9 → 32 GB, free 0.1 GB, compressor 53%) that followed three typecheck grants at 1 unit each within 35 s — tsgo is 7–10 GB at any unit count, and the pacer budgeted CPU units only; memory-guard logged "WOULD HOLD" and held nothing (watch-only). The 22:40–23:00 episode was CPU/disk (load/core 6–10, 18–32 processes in disk wait, memory fine): 28 pytest-xdist workers from two uv run pytest suites (-n auto), unseen by the pacer and by the hook's prefilter; a next dev --turbopack at 255% CPU; ~36 Claude sessions. The pacer logged 32 busy ticks across the two windows that nobody read.

Decisions (each changes how the systems relate):

  1. The pacer budgets memory as well as units. Each class carries mem_gb (typecheck 8, build 4, test/browser/render 2, lint 1) against mem_budget_gb 20; a waiter that does not fit waits, strictly, like units. Static on purpose — macOS "free" is not a signal (memory-guard's 2-of-3 bars exist for that reason); the dynamic brake is memory-guard's HOLD.

  2. memory-guard's HOLD is armed on its own (memory-guard.hold.armed): the file is raised under pressure, the pacer — its only reader — pauses new heavy work, and no process is signalled and no session refused. The runaway kill stays watch-only, as Gera asked.

  3. pytest is heavy work, and the pre-push hook is pace's second caller (pytest, uv run … pytest, python -m pytest; agent-smith's .githooks/pre-push wraps its own suites in pace --class pytest --fg --, agent-smith#738, because a suite inside git push never passes the Bash-tool hook — the first caller — and those suites were the 28 workers): 8 units, PYTEST_XDIST_AUTO_NUM_WORKERS from the grant; the dotclaude prefilter learns the words (a bare python deliberately not — every python3 - <<EOF would pay pace's start-up).

  4. Dev servers yield, never queue. A 0-unit class (next dev, vite, run dev*): exec'd in place at nice 10, no holder, no parent on a server for hours. Closes the "idle next dev" item above.

  5. Headroom for the latency-sensitive daemons is a smaller paced pool, not a priority. base_units is 12 on the 14-core mini: the normal paced pool is capped at 12 units, so paced work leaves scheduler headroom for Smith's worker, the slack socket and the sessions' own turns. Units are accounting, not CPU affinity — unpaced work, fail-open and the max-wait overflow still use every core, and pressure stays normalised by 14 — so this is headroom, not a reservation (Codex review of local-bin#362). A launchd Nice -10 on the daemons was tried the same night (agent-smith#737) and withdrawn: a 90-min A/B sampler gave a CPU burst the same 0.84 share at nice 0 and -10 at load/core 4–6 (Darwin's timeshare scheduler gives nice little leverage), and the worker is a spawn root — claude -p, start-driver → tmux, claude --resume --bg — so the priority would inherit into servers and human panes. pace's +5 nonetheless became an absolute floor (nice_target), so any future negative-nice parent still yields.

  6. Sustained saturation is loud. pace tick keeps a consecutive-busy clock and publishes status.json.busy (since, ticks, top commands, top sessions by CPU — every process charged to the registered session it descends from; no pid leaves); host-alerts posts host-saturated after 5 min of it. Signals, not calls (ADR-0003): host-alerts samples nothing.

  7. A dev server lives one hour (Gera, 2026-10-03 08:5x: "workers/sessions are most important and if something is consuming more it needs to be addressed"). The pacer's yield (decision 4) makes a server polite; it cannot end one, and at 08:46 three next dev servers were 2.7 cores and ~4 GB against 1.6 cores of actual sessions, one of them nine hours old. A new reflex, dev-reaper (local-bin#368, launchd every 60 s), stops any next dev / vite server ttlS (3600 s) after it started — a hard lifetime, not an idle rule (Gera's choice over an idle-30-min proposal; a session that still needs a server starts a new one) — and announces every stop with kind, port, worktree and age. It signals only the launcher and next's own server process, re-verified by recorded identity on every rung; never a shell, a session, a test runner or a build. Relation to the supervisor's devservers phase: that one stops only servers it created and adopts the rest; the reaper's lifetime applies to all of them, and the supervisor's reconcile pass already removes a serve entry whose port no longer listens.

Still outside the pacer's reach, on purpose: find/git grep/rg and the tsx scripts/check-*.ts pre-push checks (short and bounded — a queue wait costs more than they do); the ChatGPT/Codex app's renderers (not ours); the sessions themselves (~60–90% CPU summed across ~30 — not the load).

Amendment 2026-10-03 — the pacer reaps what has no live owner

What was wrong. Decision 6 above stamps every entry with its pid — the pid of pace itself, not of the vitest/tsgo/pytest it started. prune() therefore returned a dead holder's units and never signalled the job: a SIGKILLed pace (session killed, tmux window closed, medic relaunch) left its tree running, unaccounted and unpaced. On record: the hung shape in agent-smith's hook header (2026-09-12 — 99 % of tests reported, all 14 xdist workers at 0.0 % CPU, controller alive, log frozen eight minutes); on 2026-10-03, ~45 sessions, 231 worktrees and a 27-minute 2-core vitest run.

Decisions.

  1. The holder records its child — child_pid, the child's ps lstart under LC_ALL=C (the lease hook's one start-time format) and its argv head — in watch mode inline, in enforce under one bounded lock after the child starts (a busy lock leaves it unstamped and says so).
  2. The session's liveness decides, through the fleet's existing instruments and never a new predicate (two answers to "is this seat alive" was the 2026-09-11 incident): the conversation lease's holder (pid + start time, the rule fleet-medic relaunches on) and liveness (roster + process evidence, three-valued). A lease saying alive is alive. A reap needs both to say dead, or the primitive alone where there is no lease. A read that fails — roster timeout, unreadable ps, a stamp with no start time — is unknown and keeps the job: the predicate is a conjunction of absences (ADR-0040 review, agentflow #271). A job with no session id is never judged. The lag this carries is accepted: a session whose roster row vanished reads dead only after its transcript has been quiet 15 min with no process behind it.
  3. A live session's job is never signalled, whatever its runtime. The judging runs outside the state lock (one ps, one roster read, then dev-reaper's ladder: TERM the child → TERM the tree's survivors after 10 s → KILL after 20 s, every pid re-verified by argv head + start time before each rung), at most two reaps per tick; DISABLED disables it; pace --print-reap shows the verdicts and touches nothing.
  4. Every reap is evidence a human can falsify later: a reaped log line; a row in reaped.jsonl with session, verdict, how it was obtained, argv head, age, signals sent and survivors; one Slack line through the fleet's one poster (--source pacer, never --alert); status.json reaped_24h / last_reap. A reap-judged line on every tick — and a dated REAP-JUDGED line in reaper.log the catalog's log-match-age can read — so a tick that found nothing still proves the reaper looked (ADR-0009).
  5. Part 2, the hung job (same lane, next PR): a holder past its class's max_run_s and whose tree's cumulative CPU time has not advanced for ten ticks is reaped with why=hung; a live session's job past the maximum that still burns CPU is logged once (long-running) and left alone.