0069 — The row fence gets a warm resident client, and the hooked write path stops paying a cold connect

• Status: Accepted, and BUILT, NOT LOADED (amended 2026-11-03 and 2026-11-04; see the two amendments at the end). The client is built in agent-harness: #1904, then part 2 in #1908, #1910, #1911, #1912, #1914, #1915 and #1916 (#1909 was split into #1911 and #1912). It is not in force: the launchd job is not loaded, and guard-bash rule 15 still runs the cold row_fence_cli. Loading it and switching rule 15 are one human step (§6).

  The text below is the record as written on 2026-10-19, when nothing was built. This record exists so the decision does not die where it was made (ADR-0004: a decision that changes how systems relate is not done until its ADR is written, at decision time, by whoever was in the room). Three states are deliberately kept apart here, because collapsing them is how this row has been read as finished twice:

  1. the decision is made (this file);
  2. the build has not started, and no lane owns it — measured 2026-10-19: no SYSTEMS.md entry, no row of its own, and 0 of the 64 ADR files on main named it before this one;
  3. loading the daemon is a separate PR and a human act (infra blast radius), exactly as ADR-0058 holds for the lapse sweep.

  The board row closes on the build, not on this decision.

• Date: Decided 2026-10-18. Recorded 2026-10-19 — late by ADR-0004's standard, and the delay is itself part of the record: the ruling sat for three wakes while the row named the ADR as owed without starting it.

• Deciders: the decider rung under ADR-0027, receipt f17e06882 (Fable, claude-fable-5), verdict RESOLVED, chosen option "Build the warm resident fence client" out of three. Held under reversal; confidence medium as-given and high reversed, and this record reports the lower, per the receipt's own convention. Not a founder's decision and not this Driver's.

• Filed from: board row bug-the-row-fences-temporal-connect-cannot-finish-in-its-4s-budget-on-a-loaded-box-and-is-8x-the-board-read (dock dock-found-rows, group Found along the way), which carries the full measurement. ⚠️ That row's TITLE names the wrong mechanism — see What the title gets wrong.

• Systems: row-fence (agent-smith, in-process, status: proposed) — gains a resident client in front of it. Catalog entry added in this PR as row-fence-client, also proposed, with an empty signal list, because nothing is running yet and an entry declaring liveness it cannot prove is the defect ADR-0009 exists to stop.

The problem: the verdict is correct and arrives too late to be heard

A lane that holds a board row must pass every external write through the row fence. On a loaded box the fence refuses legitimate writes — and it refuses them fail-closed, which is the design working, not a safety hole. No write has gone through unfenced.

What is broken is availability. Measured 2026-10-18 on the mini (hw.ncpu=14), with the helper invoked directly, no hook involved:

The claim was valid and the door could not be opened in time to hear so. Control A is what makes this specific rather than "the box is slow": a bare interpreter of the same venv starts in 0.04s, so the host schedules processes fine. Of the 20.23s run, total CPU is 1.4s — roughly 19s is waiting.

The helper's bound is three constants, not a tunable:

row_fence_cli.py:130  HOOK_CHILD_DEADLINE_S = 6.5
row_fence_cli.py:136  VERDICT_MARGIN_S      = 1.0
row_fence_cli.py:193  DEFAULT_TIMEOUT = HOOK_CHILD_DEADLINE_S - VERDICT_MARGIN_S   # 5.5s

(cited at agent-smith@2352f91c875a:src/agent_smith/row_fence_cli.py, byte-identical to the copy the hook actually runs — sha256 f245b3386ba4dace… both sides.)

It is a three-way race, not a threshold and not a wall

Two earlier framings of this were wrong and are retracted in the measurement file rather than left standing. The hooked path races three clocks:

• the hook's budget, timed from spawn — observed 4262–5520ms handed to the helper across twelve refusals;
• the import, 3.5–4.7s (control B), which runs before the helper's own timer starts;
• the connect + lease ask, ~0.5s at best and ~16s at worst (20.23s minus the ~4.5s import).

So the hook hands ~5s, the import eats ~4.5s, and a connect whose best case is ~0.5s gets 0–2s. A hooked write lands only when a fast import and a fast connect coincide. Load moves two of the three, which is why the outcome tracks load without being a function of it: at 1-min load 27.17 the default bound was met 2 of 2; at 32 and above it was missed 8 of 8. That is a predictor of the odds, not a cutoff — and the row's original reading ("probabilistic, work can land on a retry") was right.

The load-independent part is the import profile. python -X importtime attributes 1.851s cumulative to row_fence_cli: asyncio 1.255s (of which asyncio.base_events 1.019s), argparse 0.356s, agent_smith.row_fence 0.194s — in a CLI the hook calls with fixed arguments.

Why no constant fixes it — written down by the module's own author

row_fence_cli.py:186-192, above DEFAULT_TIMEOUT:

⚠️ AND THIS IS A MITIGATION, NOT THE FIX. It widens the band of loads a legitimate push survives; it does not stop the verdict depending on a cold connect finishing inside a fixed bound, because no constant can. The shape that does is a warm or persistent client so the push path never pays the import at all — a new long-lived process on a shared box, with a SYSTEMS.md entry and liveness signals, which is its own row and its own decision.

One constant re-size has already been spent. The author left the remedy as a decision; this is that decision.

What the title gets wrong, and what the gateways really do

The row's title blames the Temporal connect inside a "4s budget". Both halves are loose. The budget is 3763–6001ms, not 4s, and the connect is only one of the three racers. The row's prose carries 5.5s while its title says 4s for exactly this reason, and the ADR is the place that reconciles them.

And the gateways are not fast — they are UNBOUNDED. One gateway read, driver-workflow assigned, returned rc=0 in 85.39s at the same load. updater, pr-merge and bin/docs add reach a verdict because nothing kills them at 5.5s, not because the connect is quick for them. This matters operationally: it is why a dock can still close rows (at 1–5 minutes a call) while every hooked fenced verb — git push, gh pr create, gh pr merge, gh issue edit — is dead at load ≥30.

The decision

1. Build a warm resident fence client so the hooked write path never pays the import or the cold connect. The hook asks a process that is already connected; the per-invocation asyncio/argparse/SDK import disappears from the critical path.

2. Cache the CONNECTION, never the VERDICT. A cached verdict is a lease decision served after the lease may have moved, which is the fencing generation defeated by its own accelerator. Every ask still reaches the lease.

3. Client-down REFUSES; it never falls through. The three states rule 15 of guard-bash keeps apart must survive exactly, named as the guard names them:

   • NOT ROW-BOUND (exit 0) — no descriptor resolves, so there is nothing to fence: ALLOWED, silently. This is almost every session on the machine, and refusing it would stop every git push in the fleet.
   • REFUSED (exit 3) — the lease said no (stale generation, different holder, expired lease, an item this lane does not hold): DENY.
   • COULD NOT READ (exit 1) — the descriptor or the lease could not be read or asked: DENY, because "an error reading the fence is not permission to proceed".

   And the two states that are not verdicts must keep their current handling: CUT OFF MID-ASK (the helper printed row-fence: asking — and produced no verdict) DENIES with a retry; NEVER RAN (no row-fence: line at all) denies when the lane's descriptor visibly exists and otherwise fails open with a loud warning — decision f2a1da9e1, whose reasoning is that "the instrument is broken" is not "the answer is no", and that the populations are not comparable. An unreachable resident client is COULD NOT READ, not NOT ROW-BOUND. Getting that wrong turns "fence down" into an unfenced write, which the receipt names as the severe arm of its cost-if-wrong.

4. One spelling of the fence. The resident client answers from row_fence.gate(), the same code path the gateways use. "A second spelling of a fence is a second fence, and the two would disagree on the day it mattered."

5. It enters the catalog with liveness signals (ADR-0009), and its entry stays proposed with no signals until something actually runs.

6. Loading the daemon is a human act, in its own PR.

Precedent, so this is not a new shape: ADR-0034's pacer is a launchd daemon called from guard-bash rule 28, so hook → resident-process is established here. ADR-0054 forbids daemon-hosting a fleet session; a service daemon that answers a question is not a session, so it does not bind — but read that ADR before implementing, because it is the nearest rule and the distinction is the whole reason this is allowed. The decider found no contrary ruling in the ledger.

What this ADR deliberately does not do

• It does not build the client. A new long-lived process on a box ~25 sessions share, in another repo, with an unfenced-write failure arm, wants its own lane and its own review. The Driver who took this decision did not start it, and says so rather than leaving the omission to be discovered.
• It does not load it. See Status.
• It does not retune DEFAULT_TIMEOUT. There is no sanctioned knob on the hooked path: raising SMITH_ROW_FENCE_TIMEOUT changes nothing there, because the hook kills the helper at its own _fence_left_ms regardless.
• It does not change rule 15. The three states above are the contract the client must satisfy, not a redesign of it.

What is not settled

• Whether Temporal Cloud was independently degraded on 2026-10-18 is still open, and the measurement file says so. It does not change the remedy: the import alone is 3.5–4.7s against a ~5s handed budget, so even an instant connect loses the race at load ≥30. It would change the urgency.
• The ~16s was never split between the connect and the lease query; each prints one line for both.
• Sample sizes are small. n=1 for the 20.23s ALLOWED verdict, n=1 for the 85.39s gateway read, n=2 for the only ALLOWED-at-default readings. Only controls A (×3) and B (×2) repeat. Every figure is a reading at one load on one box, and none is a constant.
• No upper bound was established for a gateway. 85.39s is the longest observed; "unbounded" is an inference from it reaching a verdict where the 5.5s helper could not, not from reading its timeout code.
• Cost if wrong, carried from the receipt: if the client is misdesigned, an unreachable or stale resident client whose absence reads as allow turns "fence down" into an unfenced write. If the diagnosis is wrong and load drops back, the fleet carries a new long-lived process and its supervision surface for a problem a quiet week would have hidden — though the measured race would recur at the next sustained spike, and the 2026-10-15 incident at load 137–142 shows this is not a one-day event.

Amendment — 2026-11-03: built, not loaded

The board row row-fence-answers-in-budget (dock send-door-registry-derived) built this decision, on decider receipt fd0bcfac7 (Fable, high), which re-confirmed that ADR-0069 had already chosen the build and ruled: amend this ADR and the existing catalog entry rather than writing a second of either; loading stays a human act; and an unreachable or slow client answers exit 1 could not read, never allowed, on one path with no fallback.

What was measured first (2026-11-03, load 81-93 on 14 cores): a cold row-bound check cost ~0.4s of CPU (the SDK import plus a fresh TLS connect), stretched to 2.06-5.70s of wall time against the 5.5s bound, and 1 run in 5 was refused at load ~65. Through the resident client, the same asks took 0.32-1.30s end to end, and all of them were allowed. What is left of a warm ask is the lease update itself. Temporal Cloud was not the slow part: an RPC on an already-connected client took 0.08-0.17s.

What is built, and how it keeps each decision:

• §1 and §2. row_fence_resident is a launchd daemon (template deploy/co.propflow.row-fence-client.plist). It holds one warm Temporal connection and answers over ~/.claude/jobs/row-fence-client/sock. It caches the connection, never a verdict, so every ask reaches the lease. The hook-side asker, row_fence_client, imports no Temporal SDK; a test pins that.
• §3, the three states. NOT ROW-BOUND is decided by the asker, from its own stdlib read of the descriptor, so a dead resident client never stops a lane that holds no row. A descriptor that cannot be read is refused by the asker itself, in gate's own words (#1908). An absent client, one that answers anything other than a verdict, or one that does not answer in time is COULD NOT READ: exit 1, fence_unreadable, recorded in the ledger. The asker prints row-fence: asking before it asks, so a hook that kills it mid-ask reads CUT OFF MID-ASK and denies.
• §4, one spelling. The resident client answers through row_fence_cli.answer and then row_fence.gate, the cold CLI's own path. The caller's environment travels as an argument (gate(env=)). It is never applied to the daemon's os.environ (#1909). A daemon running older code does not answer: every ask carries a code stamp, and on a mismatch the daemon refuses and exits, so KeepAlive starts it on the new code (#1910).
• §5, signals. This amends §5's "stays proposed with no signals until something actually runs". The entry stays proposed, but it now declares its two signals. Both now exist in code, and both read down until the human load: launchd label co.propflow.row-fence-client (the process), and a file-age check on ~/.claude/jobs/row-fence-client/heartbeat (the connection). The heartbeat is written every 30s, and only after a Temporal call succeeded, so a daemon that is loaded but disconnected goes stale. This follows the lapse sweep's precedent in SYSTEMS.md: declared, proposed, and reading down until scheduled. It flips to active once loaded and once it has answered one real fenced write.
• §6, loading. Not done by any agent. No live hook on the box was changed.

The one human step. It loads the job and points rule 15 at the asker. The job is loaded FIRST, so the hook never asks a socket that is not there yet:

sed -e "s#__VENV_PYTHON__#$HOME/code/PropFlow/agent-smith/.venv/bin/python#g" -e "s#__SMITH_HOME__#$HOME/code/PropFlow/agent-smith#g" -e "s#__HOME__#$HOME#g" -e "s#__PATH__#$HOME/.local/bin:/opt/homebrew/bin:/usr/bin:/bin#g" ~/code/PropFlow/agent-smith/deploy/co.propflow.row-fence-client.plist > ~/Library/LaunchAgents/co.propflow.row-fence-client.plist && launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/co.propflow.row-fence-client.plist && sed -i '' -e 's#-m agent_smith\.row_fence_cli #-m agent_harness.row_fence_client #' -e 's#/src/agent_smith/row_fence_cli\.py"#/src/agent_harness/row_fence_client.py"#' ~/.claude/hooks/guard-bash.sh

Rollback, also one step. It reverts rule 15 FIRST, because a row-bound hooked write would otherwise be refused "could not ask" while the job is gone:

sed -i '' -e 's#-m agent_harness\.row_fence_client #-m agent_smith.row_fence_cli #' -e 's#/src/agent_harness/row_fence_client\.py"#/src/agent_smith/row_fence_cli.py"#' ~/.claude/hooks/guard-bash.sh && launchctl bootout gui/$(id -u)/co.propflow.row-fence-client && rm ~/Library/LaunchAgents/co.propflow.row-fence-client.plist

Each sed matches exactly one line of guard-bash.sh at the time of writing (the -m invocation and the _fence_mod path). Applying the flip and then the rollback to a copy gives back a byte-identical file, and bash -n passes on it.

Still open: (as of 2026-11-03; the next amendment closes the first item)

• Head-of-line blocking. The daemon answers one ask at a time. A retrying lease can hold the lock for up to the 5.5s bound, and an ask that waits past its own deadline is refused "busy", never allowed. Two things force the lock: row_fence's module-level stage slots, and redirect_stderr, which is process-wide. Both need to become per-ask before the lock can go. That work is the next slice of this row. It narrows a refusal window; it does not affect the fail-closed contract.
• The figures above are one box at one load, and none of them is a constant.

Amendment — 2026-11-04: part 2 complete, still not loaded

All of part 2 is merged in agent-harness. Loading stays the human step (§6). No agent loaded the job or edited the live hook.

• Head-of-line blocking is resolved (#1914). The daemon now answers asks concurrently, so one slow lease no longer holds up another lane. Its two reasons for one-at-a-time became per ask:

  • row_fence's progress slots are bound per context by reset_stage();
  • verdict lines are captured per ask, behind one process-wide stderr proxy, instead of with redirect_stderr.

  A thread handed work during an ask now runs under a copy of that ask's context (#1916).

• The caller's environment is an argument (#1911): gate(env=), and the daemon never writes its own os.environ or calls chdir. A refusal the daemon records itself is filed under the caller's session and the caller's paths, never its own (#1912, #1915, #1916).

• A daemon running stale code answers nothing (#1910, #1915):

  • every ask carries a code stamp, which covers the package's .py files and the board registry directory;
  • a mismatch caused by its OWN files changing makes it exit, and launchd's KeepAlive starts it on the new code;
  • a file changed while it was loading makes it exit before it binds.

• A requirement on the human step: the asker and the daemon must load the same files. The plist runs the daemon from the deploy clone (~/code/PropFlow/agent-smith), and the switch in The one human step above makes rule 15 run the asker from that same clone: guard-bash.sh sets $_DEPLOY_CLONE to that path and runs the helper with PYTHONPATH="$_DEPLOY_CLONE/src". An asker run from a lane worktree's own .venv would have every ask refused as "runs different code".

• The load, switch and rollback commands above are unchanged. They were re-checked on 2026-11-04, after agent-harness #1916 merged (7045100b03c), on scratch copies only:

  • The plist: rendered from main, plutil -lint OK, and not loaded.
  • The rule-15 switch, on a copy of the live guard-bash.sh: exactly two lines change (_fence_mod and the -m invocation), and bash -n passes. The rollback gives a byte-identical file, and the live file's sha256 was unchanged.
  • The switched asker: a lane holding no row gets exit 0 UNFENCED. A lane holding a row, with no daemon running, prints row-fence: asking and then gets exit 1, fence_unreadable, with one refusal-log line. It fails closed.

  The commands use BSD/macOS sed -i '', because they are written for the mini.