Systems catalog
Rendered from SYSTEMS.md in the agentflow repo — the registry the Systems tab evaluates live. If a system is not on this page, it does not exist: check here before building a tool, and add an entry (with liveness signals) when you ship one. Each section carries the catalog entry, how its liveness is read, the prose beside it, and an appendix of everything linked from it.
This file is the registry. If a system is not in here, it does not exist — check this file before building a tool, because the failure mode this catalog exists to kill is two sessions independently building the same thing (the disease Spotify built Backstage's Software Catalog for; this is the same pattern at fleet scale, and it sits alongside PropFlow's existing metrics catalog and components catalog — one consistent answer to "what exists, who owns it, is it alive?").
Rendered live at propflowai.co/agents → Systems tab: the relay's
GET /api/systems parses this file and evaluates each entry's signals
against the machine's real state (file ages, log lines, launchd jobs), so the
page always shows derived truth — nobody "updates the page". Editing this
file in a reviewed PR is the only way a system enters, changes, or leaves
the catalog.
Format: each ## section is one system, carrying one fenced json block the
relay parses. Prose around it is for humans. Fields:
id— stable slug (the machine key; never rename casually)job— one sentence: what it does for the fleetkind—reflex(acts locally, no dependencies upward) |orchestration(drives goals, opt-in) |actor(an agent identity that uses the fleet) |pipeline(scheduled automation, or a check invoked by hand against a stated cadence — fleet-e2e is the member that is run, not scheduled)status—active|superseded|proposed. A superseded system stays listed withsupersededByuntil its removal ADR completes — legacy is a catalog state, not a vibe.signals— how the relay proves it's alive. Types:file-age(path + staleAfterS),log-match-age(last line matchingpatternmust be recent),launchd(label must be loaded and not crash-looping),self(the relay reporting on itself). An empty list renders as UNKNOWN, never as healthy — not proving liveness must never look like proving it.
Decision records live in docs/adr/ in this repo. A decision that changes
how systems relate is not done until it is written there — decisions made in
conversation die in conversation (learned 2026-08-29, the tmux→SendMessage
decision that existed nowhere).
nightly eval queue
| What it does | Ran the maintenance, turnover and morpheus benches against the prod bench property every night, serialised so they could not race |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | nowhere since 2026-11-01 — the bench steps were retired from Temporal tools-prod :: NightlyQueueWorkflow (its other steps still run) |
| Status | superseded → the golden-eval runner (propflowai ADR-0153 amendment 2026-11-02, migration 3): POST /api/admin/dev/evals/golden → goldenEvalWorkflow; nothing runs the benches nightly since 2026-11-01 |
| Live on the Systems tab | propflowai.co/agents → Systems → nightly eval queue |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
⚠️ SUPERSEDED 2026-11-02. The nightly bench steps were retired on 2026-11-01 (Gera stopped the nightly Willows runs), so this entry no longer claims to be alive. Everything below is the history of how it ran.
This entry exists because the catalog's silence nearly retired four working
detectors. On 2026-09-02 decision b88033792 proposed retiring
turnover-eval-daily, maintenance-eval-daily, morpheus-nightly-daily and
git-hygiene-main-drift as "not scheduled since July". Every instrument agreed:
no plists, nothing in launchctl, nothing in ~/.claude/jobs (the registry was
retired 2026-07-04), and nothing in temporal schedule list. All four were
running the whole time.
They moved off launchd into NightlyQueueWorkflow in the tools-prod
namespace in early July — propflowai-smith/config/automations.toml is the
source of truth, and its own comment records morpheus's standalone 05:00
LaunchAgent being retired on 2026-07-12 when it joined the queue. The queue
exists to SERIALISE them: as separate launchd jobs they raced on the
appfolio-45 bench. Reinstalling the plists would double-run them nightly
against the same property.
⚠️ temporal schedule list DEFAULTS TO THE APP NAMESPACE. --namespace tools-prod returns Request unauthorized from an ordinary shell, so an
unauthorised read renders as an empty list and then as "not scheduled". That is
could not look collapsed into no — the error liveness exists to forbid,
arriving through a scheduler CLI instead of a roster.
⚠️ git-hygiene-main-drift is NOT a job. It is a detector riding
MorningQueueWorkflow step 0 with review_by = "permanent" and no cron of its
own. "Reinstall it" and "retire it" are both category errors.
Why only turnover carries signals, and why that is deliberate. The two
turnover legs end their logs with VERDICT: ALL GREEN at a FIXED path — a word
that means success, which is what ADR-0009 requires. Morpheus and
maintenance-eval write date-keyed outputs
(~/Library/Logs/morpheus-nightly/<ISO>-<scenario>.log,
/tmp/maintenance-eval-state/golden-<date>/), so there is no stable path to
match a verdict against. A file-age over those directories would prove only
that something wrote a file — and morpheus is currently failing (grid RED
on gen-i3-pivot-ambiguity-sink-leak-pooling, 2026-09-02), so that signal would
paint a green row over a real regression. That is precisely the lie ADR-0009
exists to prevent, so the gap is named here rather than papered over: giving
those two benches a stable verdict file is the work that would earn them a row.
Appendix
- https://github.com/PropFlow-Technologies/propflowai#readme — the README (no docs declared in SYSTEMS.md yet; add some)
golden-eval worker
| What it does | Polls the propflow-golden-eval queue so a golden eval started from the evals page or the CLI has somewhere to run; the pollers on the two isolated maintenance eval queues exist only while a run is in flight (propflowai #24768); it never starts a run itself |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | launchd co.propflow.golden-eval-worker on the mini (RunAtLoad + KeepAlive, no schedule), installed by propflowai scripts/golden-eval-worker/setup.sh; runs scripts/temporal/golden-eval-worker.ts from the runtime worktree ~/code/PropFlow/.propflow-golden-eval-runtime, synced to origin/main on every start, and restarted hourly after a merge by launchd co.propflow.golden-eval-worker-redeploy (propflowai #24813); Temporal namespace propflow-prod.hmqma |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → golden-eval worker |
Liveness signals
| Signal | How it is read |
|---|---|
| worker loaded | launchd · co.propflow.golden-eval-worker |
The worker side of the golden-eval runner (propflowai ADR-0153 amendment
2026-11-02, migration 3): POST /api/admin/dev/evals/golden starts
goldenEvalWorkflow, and this process runs it. It only polls. The bench is the
Willows, so while Willows testing is OFF every start is refused before a
workflow exists and again inside the activity.
Deploy = restart: launchctl kickstart -k gui/$(id -u)/co.propflow.golden-eval-worker
(run.sh fetches and resets its runtime worktree to origin/main, and runs
npm ci only when package-lock.json changed). Rollback, one step:
launchctl bootout gui/$(id -u)/co.propflow.golden-eval-worker.
Kept on main by co.propflow.golden-eval-worker-redeploy (propflowai #24813): hourly,
it runs the mini's ~/.local/bin/after-pull-restart (the mofu-pull mechanism)
against the primary propflowai checkout that propflowai-main-pull keeps
current, and kickstarts the worker when main moved with a code change, the
checkout is on main and no golden run is Running on propflow-golden-eval.
It never starts a run. Baseline:
~/.claude/jobs/golden-eval-worker-redeploy/after-pull-restart.sha; log
~/Library/Logs/golden-eval-worker-redeploy.log. Rollback:
launchctl bootout gui/$(id -u)/co.propflow.golden-eval-worker-redeploy.
Proof it is polling, without a run: temporal task-queue describe --task-queue propflow-golden-eval lists a poller with identity
golden-eval-runner.
Appendix — everything linked from this entry
host headroom + session cap
| What it does | Samples whether the Mac mini can still host the fleet — free memory, compressor, swap, disk (ADR-0037: PRESSURE past 85% used), and the fleet's own footprint — REFUSES a new session past the concurrency ceiling, asks sessions idle past the window to close themselves, and lets a PERSON reap a tmux session whose host already EXITED (`session-cap reap`: every pane dead, or a bare login shell with no children, no client and no live host for its sid — ADR-0040 §4). The 300 s scheduled reap was retired 2026-10-10 (Gera 2026-10-10, fleet cleanup step 2): it was never armed and never removed a session. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/host-headroom, ~/.local/bin/session-cap, ~/.local/bin/session-exit |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → host headroom + session cap |
Liveness signals
| Signal | How it is read |
|---|---|
| sampler scheduled | launchd · co.propflow.host-headroom |
| expiry scheduled | launchd · co.propflow.session-expire |
| sampler reporting | log-match-age · ~/.claude/jobs/host/host-headroom.log · matches HOST · stale after 30 min |
| snapshot fresh | file-age · ~/.claude/jobs/host/snapshot.json · stale after 30 min |
Detail document: ~/.claude/jobs/host/snapshot.json (the Systems tab opens this row into it).
Why the pattern is the literal "HOST " (2026-09-03). The evaluator
matches log patterns as SUBSTRINGS, not regexes — HOST (OK|PRESSURE) matched
nothing and the row read down for a sampler that was writing every 5
minutes. Every sample line begins HOST , OK and PRESSURE alike, which is
exactly the liveness question this row asks.
Why the liveness signal matches HOST PRESSURE too (2026-09-03). The
sampler's OK/PRESSURE split is the WARN tier (swap >2G, compressor >25%, free
<4G) — bars a 64G mini crosses in ordinary 30-session operation, by
session-cap's own calibration (its DANGER tier, measured against the real
panic, is compressor ≥50%, swap ≥16G, free ≤1G, 2-of-3, and it refuses starts
there). Keying this row on HOST OK alone rendered the system "stale" —
which reads as the instrument broke — for every busy half-hour: on
2026-09-02/03 it sat red for a day of legal load. Stale now means what it
says (the sampler stopped writing); danger-tier pressure is the bake's job,
which reads the same log against the cap's own bars.
session-cap reap's second fact (2026-10-03, ADR-0040 §4, local-bin #372).
Measured that morning: tmux op-sess-propflow-ef — its Claude host had exited
(claude --resume 3de5e881… offered, SessionEnd marker prompt_input_exit),
the pane sat at a bare zsh with zero children and no client, the sid had no
lease; pane_dead is 0 for a shell, so the first predicate could not see it.
A session is now also reapable when every pane's process is a login shell
with zero children, no client is attached, the sid read from the pane's stamp
(session-id --sid-of-pane, ADR-0038 — and from nowhere else) has no live
host by the fleet's one liveness predicate (fleet-medic host_verdict: lease
holder pid + start time, or a process resuming the sid), and fleet-medic would
not resurrect it (its own watch/seen/deliberate rule, read from its files).
Idle time (≥ 30 min since session_activity) is a dampener only, never the
trigger; any term that cannot be read makes the session UNKNOWN — counted
(REAP unknown=N) and kept — and a run that could see nothing writes REAP BLIND why=…. Every reap is a log line, a row in
~/.claude/jobs/host/session-cap.reaped.jsonl (sid, how it was read, how
liveness was decided, pane pid/command, child count, idle seconds — enough to
falsify it later) and a smith-slack-post --source session-cap --channel #agent-smith. Idle is the later of session_activity and window_activity
(an agent-made tmux new -d session is never attached, so session_activity
alone stays at creation). A kill tmux does not confirm is REAP FAILED — no
row, no post — and reap exits 1; BLIND exits 2. First real run, 2026-10-03
11:38: REAP removed op-sess-propflow-ef — host exited (sid 3de5e881), shell idle 3569m. Since local-bin #393 co.propflow.session-cap-reap runs
session-cap reap --scheduled every 300 s and was watch-only until armed
(memory-guard's pattern): it logs REAP WOULD-REMOVE <session> … and records
the evidence row with armed: false, removing nothing, unless
~/.claude/jobs/host/session-cap-reap.armed exists — the arming is the
decision raised to Gera. Every run that judged writes REAP judged armed=0|1 … (the signal; BLIND does not match, 900 s = three missed ticks). First tick
2026-10-03 12:16:58 CDT: REAP judged armed=0 pane_dead=0 shell_would_remove=0 kept=2 unknown=0 failed=0. The liveness answer is fleet-medic host-alive <sid>
(local-bin #382): lease holder pid + start time, or a process resuming the
sid; exit 0 ALIVE / 3 DEAD / anything else UNKNOWN — not liveness's roster
0/1/2.
The scheduled reap is RETIRED (Gera 2026-10-10, fleet cleanup step 2). The arming file never
existed, so in 1,376 ticks the clock logged the same REAP WOULD-REMOVE for one
pane and removed nothing (its "reaped 1" summary over-reported a dry run). The
co.propflow.session-cap-reap daemon is booted out and its plist moved to
~/Library/LaunchAgents/retired-2026-10-10/; session-cap reap --scheduled now
REFUSES (exit 2, judges nothing), so a leftover job can never become an
unattended acting reaper. session-cap reap by hand, check and expire are
unchanged, and the REAP judged … run line (no longer armed=) is now only
written by a person's run, so it is no longer a signal.
Every other entry in this catalog describes something that restarts a part of
the fleet. This one describes the floor they all stand on, and it is here
because the floor gave way. On 2026-09-01 at 13:06 the mini kernel-panicked
after 81 hours of uptime. The panic string named watchdogd ("no checkins in
90 seconds"), which is the symptom; the stackshot named the cause: 88 Claude
Code processes holding 114.8 GB resident on a 64 GB machine, 14.9 MB of free
memory, a file cache evicted to 1.8 MB, 35.6 GB held by the compressor across
65 swapfiles, and a pageout path that asked for 3,090 pages and reclaimed 23.
Nothing could be scheduled, so nothing checked in, so the kernel panicked on
purpose. ccswitch was healing wedged sessions and rotation was spreading
credentials the whole time — every rung was working, inside a machine none of
them could see.
Session age is the variable, not session count. A fresh session is ~157 MB;
the ones that died were ~1.6 GB each, because a Claude process grows with its
context. Eighty-eight fresh sessions are harmless and eighty-eight mature ones
are fatal, which is why the snapshot carries eldestSessionHours alongside the
count.
Measuring was never going to be enough, and on 2026-09-02 Gera said so.
Decision b88294643: "Cap only: refuse to start new sessions beyond a
concurrency ceiling (~35 on this hardware), never killing a running one" — and
in his note: cap, log the refusal, do not create more until they are done, and
clean them up when they finish. session-cap (local-bin#64) is those four
clauses. start-operator, start-architect and spawn-pair all consult
session-cap check before tmux new-session; session-cap reap returns slots
by removing tmux sessions whose panes are ALL ALREADY DEAD.
⚠️ It must never touch a running session. The reaper was the rejected
option, and it was rejected for a reason with a date: on 2026-08-08 this fleet's
idleness detection misread a transient probe failure and retired 7 of 10 loops
whose tasks were still active. reap's predicate is pane_dead — tmux
reporting a process has EXITED, a fact rather than an inference.
⚠️ An undeterminable count ALLOWS, loudly, and this is deliberate. Same
three-valued rule as liveness: "could not look" is not "no". A cap that failed
closed on its own bug would wedge every spawn path on the machine — including
the session someone would start to fix it. The panic cost one reboot; a wedged
fleet costs every session. The spawn sites branch on exit code 1
specifically, never on plain command failure, because if ! session-cap check
reads exit 2 (could not look) as a refusal and reintroduces exactly that.
There is deliberately no new signal row for the cap. Its log
(~/.claude/jobs/host/session-cap.log) records refusals, and a refusal is a
NORMAL, healthy outcome — a log-match-age over it would colour the row red
precisely when the cap was working. The honest reading of "is the cap alive" is
the same one that answers "is the host alive": the sampler's HOST OK line.
HOST (the sample prefix) is the load-bearing string, and that is a
liveness choice. Until 2026-09-03 this signal matched HOST OK alone, so
a degrading-but-reporting host (lines flowing, all of them PRESSURE) went
red here — and a day of ordinary WARN-tier load rendered as stale, the
word that means the instrument broke (review verdict on #105, landed as
#106). Now this row answers exactly one question — is the sampler alive and
writing — and degradation is judged where the bars are calibrated: the
session-cap DANGER tier refuses new starts, and fleet-bake's danger-band
rule (the cap's own thresholds, 2-of-3) puts a true panic in the
regression list by name.
⚠️ Free-memory percentage is the trap, and is deliberately not the primary
threshold. At panic inactive stood at 12.2 GB — nominally reclaimable,
enough to read as ~19% available on a machine seconds from death. The kernel
asked for 3,090 pages of it and got 23. Inactive memory is a promise, not a
reserve. The signals that moved first were swap and the compressor, so those
carry the thresholds (>2 GB swapped, >25% of RAM compressed, >40 GB of Claude
RSS) and free memory is the last resort rather than the headline. Thresholds
are regression-tested against that panic's real measurements in
~/.local/bin/tests/host-headroom.py.
host-headroom measures and nothing else, and that stays true: reaping an
idle session is a judgement about whether work is worth continuing — rung 3's
question under ADR-0003, and Gera's call, never a sampler's. session-cap is
where the judgement lives, and it now has two verbs rather than one.
⚠️ reap ALONE WAS A NO-OP FOR MEMORY, AND THE DAY IT SHIPPED PROVED IT.
Its predicate is pane_dead — the process has ALREADY EXITED — so the pages it
held are already gone and the slot it returns is a tmux slot. Measured
2026-09-02, hours after the cap landed: the cap REFUSED at 36/35, 14 finished
sessions were holding 6.4 GB, and reap found nothing to remove, because
every one of those processes was still running. Closing them by hand took free
memory from 0.8 GB to 15.5 GB. Clause 4 needs something that ends a PROCESS.
So a finished session now leaves, by its own exit path. Two callers, one primitive:
agentflow-supervisor's exits.ts phase, for a session whose task is terminal — it ran afterreleases.ts, asked once, requested at most three times, then raisedsession-will-not-exitfor a person. Gone since 2026-10-18 with the Supervisor (ADR-0067); only the caller below remains.session-cap expire(launchd, every 30 min), for the sessions no task names — 48 hours of transcript silence, tmux inactivity and session age, all three, with the roster only ever able to VETO.- Both call
session-exit <sid>, the single implementation of how a session leaves:claude stopfor background,/exitin the pane for interactive. Getting that split wrong is silent in both directions, which is why there is exactly one copy of it.
⚠️ THIS AMENDS b88294643, WHICH DECLINED A REAPER — see agentflow ADR-0010
for the full argument. In short: nothing here signals. Every exit is the
session's own, so Stop hooks run, the transcript flushes and the conversation
resumes (verified live 2026-09-02 — a 355 MB session stopped, its transcript
GREW on the flush, and claude --resume returned with full context). Nothing
touches a session whose task is still open, awaiting-human included. And the
clock is a TIMESTAMP, never a status word: on 2026-08-08 a transient probe
failure produced a plausible wrong STATUS and 7 of 10 live loops were retired
in one burst, so here a failed read yields no reading, at most 3 sessions
expire per run, and if more than a quarter of the fleet qualifies at once the
run expires NONE and says so.
⚠️ expire FIRED ON NOTHING FOR ITS FIRST FIVE DAYS, AND THE REASON WAS THE
CLOCK, NOT THE FLEET. The old note here read the zero as inconclusive — 28
live sessions on 2026-09-02, longest silence 4.5 h, the box only up 27 hours.
That explanation was wrong, and the instrument said so once anyone counted:
223 runs, one session expired, ever.
transcript_age_h() asked stat for the transcript's mtime, and something
on this box touches those files without appending to them. Measured 2026-09-07:
5581af18 had gained no record in 114 hours while stat reported a write
215 seconds earlier; nine of the 49 held sessions read wrote 0.0h ago for
that reason, and 0eff9218 — silent 55 h — was already past the very window it
was being held under. A 48-hour threshold cannot be crossed by a clock that
something resets every few minutes. Fixed in local-bin#83: the clock is now the
last timestamped record, which ctx.stale_hours() had always read — one
instrument where there had been two that disagreed — the same consolidation
session-id (local-bin) made when nine fleet tools each carried their own
answer to "who am I" and three consecutive calls returned three strangers.
First honest dry run surfaced exactly two abandoned sessions (121 h and 62 h)
while the six that had been lying now report honest ages of 2–21 h and
correctly stay held.
A VETO MUST STAY CORROBORATED TO KEEP VETOING (local-bin#84). The clock was
one of four latches. The other three shared a single cause: they ran BEFORE the
clocks, so none was ever checked against evidence. Measured 2026-09-07 — four
sessions held on a roster waiting set 120–143 h earlier, eight on
stand-down markers up to 119 h old (26 exist, the oldest 457 h; the
marker carries no expiry of its own), seven on awaiting-human tasks nobody had
touched in 93–158 h. A marker with no expiry is not a person holding a
session; it is a file nobody deleted.
The three clocks are now read first and the latchable vetoes are asked to agree with them. ⚠️ This does not make a status word a trigger — the 2026-08-08 inversion holds, nothing expires because of a status. All three clocks must independently be old before a stale veto stops counting, and the verdict names what it ignored, so the log never drops a hold silently:
DRY would expire 367e7813 — idle 120h (last turn), 122h old, nobody attached;
stale holds ignored: stand-down 121h old
What stays absolute, and no clock outvotes: a session pinned in the
Agent view (pins.json is the deliberate keep, and the one to use for a hold
that should outlive any window), undrained mail, an attached tmux
client, recent tmux activity, an absent status, and a task in active or in
a state this tool does not recognise. Among task states only awaiting-human is
de-latched — it is the one state where no agent acts without a person, so a warm
process buys nothing a resume would not restore, and expiring does not touch
the task, which stays awaiting-human on the board exactly as it was.
A CLOCK MUST STILL BE TRUE AT THE MOMENT YOU ACT ON IT (local-bin#85).
Selection and action were separated by a roster read — which has taken the full
30 s and timed out — plus a transcript read per candidate, with nothing
re-checked in between. Measured 2026-09-07: c9a4b20d was 122 h idle when it
was selected, a nudge wake landed in the gap, and /exit was typed at a session
that had begun working seconds earlier. Nothing was lost, because a busy
composer swallows the keystrokes — which is also why the run logged ISSUED for
a session that never left. The session's own clock is now re-read immediately
before the exit is issued, and one that has spoken since selection is stood down
for that run:
EXPIRE STOOD DOWN c9a4b20d (bin-91) — it took a turn 0.0h ago, inside the 48h
window; not typing at a working session
An unreadable transcript at that moment counts as woken, not as permission.
The summary line carries woke=N, so a stood-down session is never silent.
session-cap expire -n prints which hold spared each session, and is the
instrument for all of it.
And the brake can blind itself under load. Five runs logged EXPIRE UNKNOWN roster unreadable (claude agents --json timed out after 30s) — the fuller the
fleet, the slower the roster read, so the pressure that makes expiry necessary
is what stops it happening. Failing to no-reading is correct; needing to is the
finding.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
Fleet stability check
| What it does | Every 2h, reads what host-headroom and session-cap already measured, adds the four checks the 09-17 spawn storm proved were missing, and posts to Slack when something is wrong |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fleet-stability |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Fleet stability check |
Liveness signals
| Signal | How it is read |
|---|---|
| check scheduled | launchd · co.propflow.fleet-stability |
| check reporting | log-match-age · ~/.claude/logs/fleet-stability.log · matches FLEET · stale after 360 min |
This exists because the instrument was right and nobody heard it. On
2026-09-18 08:02 the mini kernel-panicked for the second time that month, on VM
compressor segment exhaustion. host-headroom had been reporting
HOST PRESSURE continuously since 2026-09-16 21:31 — 34.5 hours — and
sampled last at 07:53:36, eight minutes before the panic. Nothing was wrong with
the detector. Measured the same day: no path on this box joined it to Slack, a
canvas, or any other human-facing surface. It wrote a log nobody opened.
So this member adds no measurement of the host. It reads host-headroom's
snapshot — its state and reasons verbatim, so the panic-calibrated
thresholds live in exactly one place and cannot drift into a second copy — plus
session-cap's live= series. On top of that it carries the four checks the
2026-09-17 storm proved were missing: burst (the fleet grew +90 in two hours
to a peak of 134 against a ceiling of 35), roster-as-prompt (new jobs whose
intent is the session roster — the nudge:1568 bug, local-bin #231, returning),
armed copies (files still carrying the unescaped nudge backtick anywhere,
including worktrees, where it outlived the working-copy patch by a day), and
stuck-blocked (live sessions parked on a question nobody will answer, which
session-expire cannot reach because they are younger than its 48h window and
their status is not idle).
⚠️ IT ONLY READS AND REPORTS, AND IT MUST NEVER SPAWN. No killing, throttling or reaping — ending a session is rung 3's judgement and Gera's call under ADR-0003. The no-spawn rule is stricter than politeness: a monitor that can start a session is the exact shape of the bug this one watches for.
⚠️ TWO HOURS IS THE HUMAN CADENCE, NOT THE DETECTION CADENCE. The 09-17
storm went 16 to 134 sessions in 2h30m, which a two-hour poll would have caught
once. Fast detection stays with the 300s host-headroom sampler in the host
entry; this rung's only job is that a person finds out. Slack carries exceptions
plus one all-clear a day, because twelve green posts a day is wallpaper and a
muted channel is worse than no channel — and the log line is written on EVERY
run, which is what the signal above reads, so silence in Slack never has to be
taken as health on faith. staleAfterS is 21600: three missed ticks before the
row changes colour, so a reboot or a laptop lid does not read as a dead checker.
⚠️ A JOB RECORD IS NOT A SESSION. The first dry run announced "30 sessions
parked in blocked >12h holding memory" while the roster held exactly ONE live
session: 181 records said state: blocked because that word is written once and
never revised, so a rebooted box reads as a graveyard. The check now intersects
with the live roster, an unreadable roster yields unknown rather than zero, and
the regression is pinned in
~/.local/bin/tests/fleet-stability-alarms-are-not-vacuous.py — whose cases all
CREATE the bad state and assert the alarm fires, because two guards written
during the same investigation looked fine and were not.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
memory guard
| What it does | Every 30s from launchd, reads kernel memory pressure and the compressor; under pressure raises a HOLD file launchers pause on, ends a single runaway rg/fd/find/bfs/ugrep by PID, restarts a bloated mds, restarts an fseventsd past 4 GB or burning 0.75 core for 30 min past 12 h of age (SIGKILL + launchd respawn, verified, its own arm file, one restart per 6 h), and alerts — never a session count, and WATCH-ONLY until armed |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/memory-guard (launchd co.propflow.memory-guard) |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → memory guard |
Liveness signals
| Signal | How it is read |
|---|---|
| guard scheduled | launchd · co.propflow.memory-guard |
| guard ticking | log-match-age · ~/.claude/jobs/host/memory-guard.log · matches GUARD · stale after 2 min |
| snapshot fresh | file-age · ~/.claude/jobs/host/memory-guard.json · stale after 2 min |
Detail document: ~/.claude/jobs/host/memory-guard.json (the Systems tab opens this row into it).
Why it exists (2026-09-24). The third panic in 23 days was not the fleet:
at the 18:50 JetsamEvent the two largest processes on the 64 GB mini were
mds (Spotlight, indexing OFF) at 27.3 GB and ONE rg --files /Users/miniclaw
at 25.2 GB. The host entry only measures and session-cap only refuses
starts, so nothing on the box could end a runaway. Gera's constraint: protect
the machine without capping the number of agents.
Both of its mechanisms are needed, because the deaths look nothing alike.
09-18 was ~99 GB of Claude processes with no runaway at all — only the HOLD
(a pause for launchers) helps. 09-24 was two non-session processes — only
ending them helps. memory-guard --replay <JetsamEvent.ips> shows each.
The pattern is the literal "GUARD ", for the reason the host entry
records: patterns are substrings, and both GUARD OK and GUARD PRESSURE
lines prove the guard is alive. PRESSURE is shown by the detail document, not
by turning a live guard's row stale. staleAfterS is 120: four missed 30s
ticks.
⚠️ IT SHIPS WATCH-ONLY and says so in every line (WATCH-ONLY,
WATCH-ONLY+MDS, ARMED). With no flag file it acts on nothing and logs
WOULD HOLD / WOULD END pid … / WOULD RESTART mds … / WOULD ALERT.
~/.claude/jobs/host/memory-guard.mds-restart.armed arms only the mds restart
(launchd respawns mds at ~30 MB; the only thing that releases its fsevents
backlog, measured); memory-guard.armed arms everything, after ~24 h of WOULD
lines. It kills only by PID after re-reading that pid, never Claude hosts,
node, python or shells, and never pkill. Status stays proposed until the
plist is bootstrapped; the launchd signal reads down until then, which is
true.
fseventsd (2026-10-03, local-bin#406). The same shape, one more arm file:
~/.claude/jobs/host/memory-guard.fseventsd-restart.armed restarts fseventsd over
4 GB, or past 12 h of age at ≥ 0.75 core for 30 min (one restart per 6 h) — SIGKILL, because TERM is ignored (measured the day it reached 15 GB), then
a wait for launchd's respawn, the new pid verified, RESTART fseventsd old=… new=…
logged and posted. Created at deploy on Gera's approval of that day's hand restart;
rollback is deleting the file. See _fseventsd_note.
Launchers that should consult ~/.claude/jobs/host/HOLD (not yet wired):
session-cap check (covering start-operator / start-architect / spawn-pair),
ccswitch exec, pf/mpf, fleet-medic's relaunch path, and propflowai's
.husky/post-checkout worktree provisioning.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
dev-reaper (a dev server lives one hour)
| What it does | Stops any `next dev` / `vite` dev server ttlS (3600 s) after it started, whatever it is doing, and says so — logged, recorded in reaped.jsonl and announced in Slack with kind, port, worktree and age — so a forgotten server never holds a core and a gigabyte for a day |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/dev-reaper (+ dev-reaper.json; launchd co.propflow.dev-reaper, every 60 s) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → dev-reaper (a dev server lives one hour) |
Liveness signals
| Signal | How it is read |
|---|---|
| reaper scheduled | launchd · co.propflow.dev-reaper |
| status fresh | file-age · ~/.claude/jobs/dev-reaper/status.json · stale after 3 min |
Detail document: ~/.claude/jobs/dev-reaper/status.json (the Systems tab opens this row into it).
Why it exists (Gera, 2026-10-03 08:5x). "Workers/sessions are most
important and if something is consuming more it needs to be addressed."
Measured at 08:46 on the mini: the sessions themselves were 1.6 cores
(drivers 1.1, interactive 0.2, Smith 0.3); three next dev servers were 2.7
cores and ~4 GB — one nine hours old from a worktree whose work was long done,
each started by a shell with no session behind it any more. Turbopack watches
the tree whether or not anyone asks for a page. The pacer can only make a
server yield (nice 10); the Supervisor's devservers phase stopped only servers
it started (and is deleted, ADR-0067); sessions forget.
The rule is a lifetime, not idleness — Gera's decision, replacing an
idle-30-min proposal: a server expires ttlS after it started. A session that
still needs one starts a new one. Nothing dies silently: every stop is logged,
written to ~/.claude/jobs/dev-reaper/reaped.jsonl (a session whose curl just
failed can read why) and announced through smith-slack-post --source dev-reaper (never --alert).
What it signals, and what it never touches. A node-family program whose
command word is next dev or vite [dev|serve] — the outermost launcher and
next's own next-server beneath it. A shell's -c string, a session's argv,
a log line containing the words, next build/start, a test runner or build
spawned beneath a server: never. Every member's argv head and age are recorded
at discovery and re-verified on every rung (TERM launcher → TERM survivors →
KILL survivors), so a reused pid is never signalled. Kill switch
~/.claude/jobs/dev-reaper/DISABLED; dev-reaper --print touches nothing; an
unreadable ps signals nothing.
Signals. status fresh: the tick rewrites status.json every 60 s whatever
it finds, so a dead job goes stale in three missed ticks. Known gap, the same
shape as the pacer's: a tick that is alive but mis-discovers reads as "no
servers" — the discovery is pinned by the suite's fixture table, not by
production.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
hooks-pin (the mini's pre-push does not depend on the branch)
| What it does | Every 300 s, pins core.hooksPath of propflowai, propflowai-smith and agent-smith at ~/.local/bin/githooks/<repo> in each repo's COMMON git dir — one write covers every worktree — re-pinning whatever husky's `prepare` re-set and logging the value it replaced; the machine pre-push there execs the branch's own hook only when it carries `# pre-push-profile: cheap-2026-10-03`, else runs the canonical cheap set from the worktree's scripts and says so in one line |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/hooks-pin + ~/.local/bin/githooks/{propflowai,agent-smith}/ (launchd co.propflow.hooks-pin, system domain, every 300 s) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → hooks-pin (the mini's pre-push does not depend on the branch) |
Liveness signals
| Signal | How it is read |
|---|---|
| pin scheduled | launchd · co.propflow.hooks-pin |
| state fresh | file-age · ~/.claude/jobs/host/hooks-pin.json · stale after 15 min |
| tick logged | log-match-age · ~/.claude/logs/hooks-pin.log · matches tick · stale after 15 min |
Detail document: ~/.claude/jobs/host/hooks-pin.json (the Systems tab opens this row into it).
Why it exists (ADR-0040, amendment "2026-10-03 — bulletproof enforcement
in layers", layer 2). Measured 2026-10-03 11:35: propflowai's
core.hooksPath was the RELATIVE .husky/_ and agent-smith's the relative
.githooks, so a push from a worktree ran THAT BRANCH's hook. When main's
hooks got cheap (propflowai #16046 drops the local mutation-control vitest
run; agent-smith #749 keeps pytest to the cross-repo intersection), 186 of
207 propflowai worktrees and 13 of 15 agent-smith worktrees — plus every one
of Smith's /private/tmp/wt-* lanes, which share agent-smith's common git dir
— still carried the old heavy hook until each branch rebased, and nothing
rebases propflowai branches (strict=false; branch-updater skipped it, and was
retired 2026-10-10). One such lane ran the full
agent-smith suite at 1.0 core at 11:32. check-husky-runtime.ts already
accepts an absolute, existing hooksPath, so the pin reddens nothing.
What it does and does not touch. git config --file <common>/config core.hooksPath <abs dir> when the value differs — never a plain git config
from a worktree, which under extensions.worktreeConfig would pin one tree.
pre-commit and post-checkout pass straight through to the worktree's own
copies (route manifest, worktree provisioning), so pinning costs nothing it was
not meant to. A machine hooks dir that does not exist is NOT pinned (git would
run no hooks, silently) and reads as unpinned. hooks-pin --check exits 1
when any present repo is unpinned (layer 6 / host-alerts). Rollback:
fleet-daemon --rollback co.propflow.hooks-pin, then per repo git config --unset core.hooksPath (propflowai back to .husky/_, agent-smith .githooks).
Signals. tick logged: the job prints one tick … line per run whatever
it finds, so a quiet log is a dead job, never a pinned fleet. state fresh:
hooks-pin.json is rewritten every tick with checkedAt, repins and the
replaced value per repo. Known gap: a tick that is alive but points at the
wrong directory reads pinned — the directory's contents are pinned by
local-bin's test (tests/a-worktree-on-an-old-branch-runs-the-machines-cheap-pre-push.sh),
not by production.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
pacer (heavy work takes turns on the host)
| What it does | Heavy commands (tests incl. pytest, type-checks, builds, repo-wide lint, headless browsers, renders) take turns in a pool of CPU units AND a memory budget: fair share per Claude session with borrowing, parallelism set from the grant, capacity shrinking under disk-wait/load pressure, memory-guard's HOLD respected, dev servers yield at nice 10 instead of queueing, a busy tick names what the load is and whose, and every tick publishes how much of the box goes to non-session work and to the fleet's own tooling (status.json.cores), and the tick reaps a paced job whose session is DEAD (lease holder + liveness both say so) — logged, recorded, announced, and never a live session's work — never a cap on agents, and it fails open |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/pace (+ pace.json; launchd co.propflow.pacer = `pace tick`); callers: rewrite rule 28 in gera-propflow/dotclaude hooks/guard-bash.sh, and agent-smith .githooks/pre-push (`paced`, agent-smith#738) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → pacer (heavy work takes turns on the host) |
Liveness signals
| Signal | How it is read |
|---|---|
| tick scheduled | launchd · co.propflow.pacer |
| status fresh | file-age · ~/.claude/jobs/pacer/status.json · stale after 2 min |
| reaper judging | log-match-age · ~/.claude/jobs/pacer/reaper.log · matches REAP-JUDGED · stale after 5 min |
Detail document: ~/.claude/jobs/pacer/status.json (the Systems tab opens this row into it).
Why it exists (measured 2026-09-28 ~00:05 on the mini). 14 cores, ~12
sessions plus subagents, load ~30 sustained and 60 at peaks; at one moment 24
vitest processes, 17 render processes, 5 next dev, tsgo type-checks at 7–10 GB
each, eslint and headless chrome — none aware of the others. When bursts lined
up, 20–34 processes sat in uninterruptible disk wait, the ccswitch snapshot went
stale, and the kernel panicked twice that week from userspace-watchdog stalls.
Gera's constraint, as for memory-guard: no cap on the number of agents — only
heavy work takes turns. Decision record: docs/adr/0034-heavy-work-takes-turns-by-fair-share.md.
How it works. dotclaude guard-bash.sh rule 28 rewrites a heavy Bash
command to pace --class X [--fg] -- <original> through PreToolUse
updatedInput; sessions change nothing, and where pace is not on PATH (the
Mac) the command runs untouched. A second caller since 2026-10-03:
agent-smith's .githooks/pre-push wraps its own pytest suites in
pace --class pytest --fg -- (paced, agent-smith#738) — a suite that starts
inside git push never passes the Bash-tool hook, and on 2026-10-02 two of
them were the 28 python -u workers in every busy line. Same fail-open: no
pace on PATH, the suite runs as before. pace asks for units under one flock on
~/.claude/jobs/pacer/state.json: max-min fair share per
CLAUDE_CODE_SESSION_ID (a session and its subagents are one account), a session
alone may borrow the whole pool, freed units go to the waiting session holding
the fewest (ties first come), no preemption. The child's parallelism comes from
the grant (VITEST_MAX_FORKS/THREADS, GOMAXPROCS,
PYTEST_XDIST_AUTO_NUM_WORKERS) plus nice +5 as an absolute floor (a child
of a negative-nice parent still lands at +5); render/build share a long lane
capped at min(4, capacity/2). Capacity is AIMD from base_units — on the mini
12 units of 14 cores: the paced pool is capped so paced work leaves scheduler
headroom (units are accounting, not affinity; unpaced work, fail-open and the
overflow path still use every core; 2026-10-02: a Nice -10 on Smith's daemons
was measured useless, the smaller pool is the lever); null, the default
elsewhere, means every core — floor base/2, on U-state count and load/core. Memory is a second budget (2026-10-02, ADR-0034 amendment):
each class costs GB (typecheck 8 — tsgo is 7–10 GB at any unit count; pytest 3,
an estimate) against
mem_budget_gb 20, and a waiter that does not fit waits, strictly, like units.
Every number in this paragraph is as of 2026-10-02; ~/.local/bin/pace.json
is the live copy and wins when they differ. Dev servers yield, never queue: next dev, vite,
npm run dev* are a 0-unit class exec'd in place at nice 10. Every entry is
stamped with a pid and a flock its process holds, so a killed holder's units
come back on the next lock.
Modes (~/.local/bin/pace.json, read on every call): off · watch (never
waits, never touches the child — records demand for pace replay) · enforce.
It shipped watch on 2026-09-28 and has run enforce since. Kill switch: touch ~/.claude/jobs/pacer/DISABLED —
everything runs unpaced and the hook stops rewriting. It fails open on any error
and after the max wait (30 min; 75 s for a foreground Bash call, whose timeout
the hook extends to cover it). host-alerts posts when the oldest waiter passes
10 min, and — host-saturated — when the tick has seen the box busy (load/core
≥ 4 or ≥ 50 processes in disk wait) on every tick for 5 min, naming the top
commands and the top sessions by CPU from status.json.busy (2026-10-02:
32 busy ticks across two windows while Smith's worker starved, read by nobody
until morning). Two more rules read status.json.cores (local-bin#380,
2026-10-03): tooling-over-budget — the fleet's own tooling at ≥ 1.0 core
on every 60 s read for 10 min, recovering after 10 min under 0.7 (a reading in
between changes nothing), the post naming the top 3 tooling commands with
cores and the top 3 non-session processes a human reads (Codex, XProtect —
things no reflex may kill); fseventsd-hot — cores.named.fseventsd
≥ 0.75 core for 10 min, clear under 0.5, and only on the tick that posts one
bounded sudo -n sample fseventsd 2 names the hottest client (thread headers
carry the client pid; a claude is named by its session; sudo refused →
"client unknown (no sudo)"). A null or stale reading is unknown and advances
no clock. host-alerts samples nothing else (ADR-0003) — except, since
2026-10-03 18:00 (fseventsd-rss, local-bin#407), when memory-guard's snapshot
says fseventsd holds ≥ 2 GB for 10 min: one post with RSS, age, CPU and the top
FSEvents client adders from a bounded live log stream trace run only on that
tick (an hour-old log cannot be mapped by pid — pid reuse, measured), recovering
under 1 GB; memory-guard's own restart is at 4 GB or the CPU form (see its _fseventsd_note). And — local-tests-ran
(local-bin#388, ADR-0040 layer 6) — when status.json.local_tests says tests,
typechecks or builds that nothing admitted took more than 60 s of CPU in the
last hour: one post naming the last offender, its session and worktree, and
the layer that should have caught it; recovery when the hour reads 0. The
4-day test-census (expires 2026-10-07 12:00) is the temporary receipt this
permanent signal supersedes — see _local_tests_note.
The reaper (2026-10-03, under the host-throughput brief; ADR-0040 §4,
ADR-0034 amendment of the same day). Every holder entry recorded pace's
own pid, so prune() returned a dead holder's units and never signalled the
job: a SIGKILLed pace (session killed, tmux window closed, medic relaunch)
left its vitest/tsgo/pytest tree running unaccounted and unpaced — the hung
shape agent-smith's hook header records (2026-09-12: 99 % reported, 14 xdist
workers at 0 % CPU, log frozen 8 min). Measured 2026-10-03: ~45 sessions, 231
worktrees, a 27-minute 2-core vitest run. Now the holder records its child
(child_pid, ps lstart under LC_ALL=C — the fleet's one start-time
format — and the argv head), and every tick, after prune and outside the
state lock, judges each holder and each pruned entry whose child still runs by
the liveness of its session, through the two instruments the fleet already
trusts and never a new one: the conversation lease (holder pid + start time,
what fleet-medic relaunches on) and ~/.local/bin/liveness (roster + process
evidence, three-valued). A lease saying alive is alive; a reap needs both
to say dead, or the primitive alone where there is no lease; a read that fails
is unknown and keeps the job — a reap predicate is a conjunction of absences.
Gera's rule, unchanged: a live session's work is never signalled, whatever
its runtime — a 27-minute vitest run in a live session is foreground work.
Dead → dev-reaper's ladder: TERM the child, TERM the tree's survivors after
10 s, KILL after 20 s, every pid re-verified by argv head + start time from a
fresh ps before each rung (a reused pid is never touched), at most two per
tick. What it never touches: a job whose session is alive or unknown; a
job with no session id (agent-smith's pre-push suites); anything under
~/.claude/jobs/pacer/DISABLED; anything under pace --print-reap; anything
when ps is unreadable. Every reap: a reaped log line, a row in
~/.claude/jobs/pacer/reaped.jsonl carrying the evidence it acted on
(session, verdict, how obtained, argv head, age, signals, survivors), one
Slack line through smith-slack-post --source pacer (never --alert), and
status.json reaped_24h / last_reap / reap_judged_at. Shipped in
local-bin#371 and #376 (the judge: lease and liveness; an unreadable read
is unknown, #381). The hung job (#379): a holder past its class's
max_run_s whose tree's cumulative CPU time and process count have not
changed across ten ticks is reaped why=hung; a tick that cannot read a
time adds no sample, so the clock does not advance. Unpaced trees and the
hard max (#385, ADR-0040's layers amendment, layer 4): the tick also scans
the table it already reads for heavy roots it does not hold — vitest and
its forks workers, tsgo/tsc, pytest and its xdist workers, next build,
playwright, by argv shape — and walks each root's ancestry: a live claude
above it is never touched (named once, unpaced-live); a human's tmux /
login / bare shell, a pace wrapper, a program of its own (a launchd job's
bash run-all.sh) or an ancestry the table cannot resolve are kept and
counted; nothing but wrappers up to launchd is an orphan and goes down
the same ladder with the ancestry as evidence. And a job of class test /
pytest / typecheck / build / browser (reap.hard_max_classes) older than
max_run_s = 1200 s is reaped whatever its CPU and whoever owns it,
held or unpaced, live session or not (why=max-runtime) — Gera's call,
2026-10-03 11:4x, the one class of work exempt from the never-kill-foreground
rule; rollback = empty the list. The refusal layer (#389, layer 3): in
enforce, a request of class test / pytest / typecheck / build / browser /
render is refused before any child — exit 1, one paragraph on stderr, a
refused log line, refused_24h in status — unless it names at most
local_tests.max_named_changed_files (1; 0 = cloud-only) test files each
changed in its cwd branch vs origin/main (the guard's rule-16 predicate).
A request it cannot check (no checkout, no origin/main) is refused: this
layer fails closed, unlike the guard. CI_PUSH_GUARD=1 is exempt, and so is the
machine hook's cross-repo pytest set — the hook's exact form with every id
listed in ci/cross-repo-tests.txt at origin/main of that repo (bounded by
the file, not by N; admitted, admitted_24h); watch logs would-refuse
and runs; DISABLED runs; rollback = empty local_tests.refuse_classes.
Signals. status fresh is the liveness signal: pace tick rewrites
status.json at least every 30 s (and every client on grant/release), so a dead
tick goes stale in four missed ticks. Known gap: a tick that is alive but whose
busy block is broken reads as calm (since: null), and nothing proves the
saturation path end to end short of a saturated box — the same shape as the
Smith alarm's gap above; the busy-tick test covers the parse, not production. pacer.log is JSON lines with an epoch t
field, which log-match-age cannot date (it wants a YYYY-MM-DD HH:MM:SS
prefix) — so there is deliberately no log signal; the file-age one already
fails whenever the tick stops. reaper judging (2026-10-03) is the one log
signal, and it reads a different file: the tick writes one dated
REAP-JUDGED judged=N … line to ~/.claude/jobs/pacer/reaper.log on
every tick, reaping or not, so a tick that judged nothing still proves the
reaper looked and a reaper that stops judging goes stale in ten missed ticks
(ADR-0009).
The status contract — ~/.claude/jobs/pacer/status.json. Rewritten
atomically on every grant and release and at least every 30 s (pace tick).
These field names are stable; add, never rename:
| field | meaning |
|---|---|
version |
contract version (1) |
mode |
off · watch · enforce (off also when DISABLED exists) |
capacity_units / used_units |
the pool now (after AIMD) / units held |
pressure.u_state / .load_per_core / .hold |
processes in uninterruptible wait · 1-min load ÷ cores · memory-guard HOLD up |
holders[] |
{session, session_name, class, units, pid, cmd_head, since} |
waiting[] |
{session, session_name, class, units_wanted, since} |
per_session[] |
{session, session_name, units, share, waiting} — only sessions with heavy work running or waiting |
memory.budget_gb / .used_gb |
the memory budget and what holders cost (holders[].mem_gb) |
busy |
{since, ticks, load_per_core, u_state, top_cpu[], top_dwait[], by_session[]} — consecutive busy ticks and what the load is; since: null when calm. host-alerts reads since |
reaped_24h / last_reap / reap_judged_at |
reaps in the last 24 h · the last one {at, session, session_name, class, cmd_head, age_s, why, survivors, announced} or null · when the reaper last judged (epoch); why is session dead · hung · max-runtime · orphan |
unpaced |
{orphan_unpaced, unpaced_live, unpaced_human, unpaced_unknown, unpaced_paced, unpaced_young} from the last judging — heavy trees the pacer did not hold, by what the ancestry said |
refused_24h |
requests the refusal layer turned away in the last 24 h (local_tests) |
admitted_24h |
cross-repo pytest sets the refusal layer let through in the last 24 h |
cores |
{at, interval_s, session, nonsession, tooling, top_nonsession[{comm, cores}], top_tooling[{comm, cores}], named{fseventsd}, ms} — the box's CPU over the last tick interval, from CPU-time deltas; null numbers when there is no previous sample. host-alerts reads tooling and named.fseventsd |
local_tests |
{at, window_s, cpu_s_last_hour{exempt, nonexempt}, procs_seen, last_nonexempt{comm, kind, session, entry, cwd, cpu_s, at}} — local test / typecheck / build CPU in the last hour, admitted vs not. host-alerts reads cpu_s_last_hour.nonexempt |
holders[].exception |
why the refusal layer admitted a held run locally (changed-files: N · cross-repo set · ci-push-guard); null for a class it does not judge — local_tests counts the tree under it exempt |
updated_at |
epoch seconds |
Also present, not part of the stable set: system, disabled, base_units,
long_lane{cap_units, used_units}, holders[].overflow, holders[].watch (in
watch mode every holder is observed, not granted — so used_units may exceed
capacity_units), holders[].fg. Times are epoch
seconds; session is the full session id (- when the caller had none).
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
guard-kill-rate (guard-bash kills per 1,000 Bash calls, busy vs quiet)
| What it does | Read-only report: guard-bash kills (harness 10s kill) and over-budget verdicts per 1,000 Bash calls, by UTC day, split by the pacer's load/core and disk-wait ticks, so the load-spikes before/after is a command |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | ~/.local/bin/guard-kill-rate (local-bin#517); run by hand: `nice -n 19 guard-kill-rate --since <day>` |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → guard-kill-rate (guard-bash kills per 1,000 Bash calls, busy vs quiet) |
Liveness signals
| Signal | How it is read |
|---|---|
| input: pacer ticks being written | file-age · ~/.claude/jobs/pacer/pacer.log · stale after 5 min |
Run by hand, never on a timer, and it writes nothing, so it has no liveness
of its own. The signal proves its main input is alive instead: without fresh
pacer ticks every bucket reads "no pressure sample". The method is the
2026-10 kill-cause page's (docs.propflowai.co/a/guard-bash-kill-cause-2026-10):
kills are hook_cancelled attachments for "Checking bash guards" in Claude
Code transcripts, once per toolUseID, over Bash tool_use ids. Buckets are
10-minute UTC windows with at least 20 calls. Each bucket's pressure is the
max pacer tick inside it, with cells split at 4 load/core and 100 processes
in disk wait. A bucket with no tick is reported as "no pressure sample",
never as quiet. Since dotclaude #139/#143 the hook ends itself at 7.5s and
writes guard-bash-overbudget.log instead of being killed, so those lines
are counted beside the kills: a hook that stopped being killed must not read
as a quieter box. Its history is as long as pacer.log's rotation keeps
(local-bin#519 raises that to 30 MB, at least 7 days).
Appendix
- https://github.com/PropFlow-Technologies/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
ccswitch
| What it does | Credential reflexes over SEPARATE account pools, one per owner and never shared (the fleet's is propflow-claude-account-pool; the GitHub reviewers' is claude-pool-reviewer, fed to CI by reviewer-tokens) — per-session switching that interrupts (80% on the 5h/weekly axes, 95% on Fable), wall healing, headroom-aware spawn assignment that will not send a session to a seat it cannot use, wedged-session healing, and a move that REWRITES a session's credential file in place rather than killing its pane (the DEPRECATED respawn still HOLDS while the pane has live subagents); mint is the only credential door |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/ccswitch |
| Deployed at | ~/.local/bin/ccswitch |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → ccswitch |
Liveness signals
| Signal | How it is read |
|---|---|
| pool cache fresh | file-age · ~/.claude/ccswitch/cache.json · stale after 60 min |
| session registrations flowing | file-age · ~/.claude/ccswitch/reg · stale after 60 min |
| rotation telemetry | file-age · ~/.claude/ccswitch/rotation.log · stale after 2880 min |
| account snapshot fresh | file-age · ~/.claude/ccswitch/snapshot.json · stale after 5 min |
| swappable sessions being born | file-age · ~/.claude/ccswitch/session-cfg · stale after 1440 min |
| subagent-hold guard consulted | file-age · ~/.claude/ccswitch/subagent-holds/.consulted · stale after 2880 min |
| walled sweep on time | file-age · ~/.claude/ccswitch/walled.json · stale after 3 min |
Detail document: ~/.claude/ccswitch/snapshot.json (the Systems tab opens this row into it).
One pool per owner, never shared (2026-10-08, dotclaude#206). Gera: "we
don't want to share pools but we do want to have the pools somewhere." A pool
is one AWS Secrets Manager secret in ccswitch's pool layout (accounts,
roster, owners, retired), so who may read it is an IAM question, not a
convention: a machine reads only the pool it is pointed at
(CCSWITCH_POOL_SECRET, default propflow-claude-account-pool).
| Pool (AWS secret) | Owner | Runs on | Accounts |
|---|---|---|---|
propflow-claude-account-pool |
Gera's fleet (Gera's accounts; clara@propflowai.co is Fede's) | the mini fleet (ccswitch) |
ccswitch list (live) |
claude-pool-reviewer |
GitHub reviewers (Gera's accounts) | GitHub CI review and CI evals (reviewer-tokens → CI secrets) |
nana.ai.app and propflow.owner (primary, reviewers only); propflow.agentsmith, propflow.sandbox, propflow.operator, proflow.clara (not the remote pool's propflow.clara), propflow.agent, josegerag and propflow.lab (secondary, also on the mini fleet); propflow.test (evals, also on the mini fleet) |
claude-pool-remote |
Gera | AWS worker, Lambdas (SSM) and Vercel production, all through remote-pool |
josegarzadev@gmail.com and propflow.clara@gmail.com, only (both left the fleet's pool 2026-10-28) |
Where every account runs, live. Each pool table on the ccswitch row of
/agents has Owner and Runs on columns (the live headers), read from the
placements map that remote-pool-cycle.py (placements()) writes each cycle. Every place an account's login is in use
is listed: the mini fleet, GitHub reviewers (CI), AWS worker, Lambdas (SSM),
Vercel production. Fede's logins are not used on our servers (Gera,
2026-10-28). They live in Fede's local pool. The one in the mini's bank
(clara@propflowai.co) stays there by Gera's call. If one ever shows up on a
server copy, the remote pool's table names it in red (that check is
fedeOnServers in propflowai's accountPlacements.tsx, over the same map).
An account moves between pools by ccswitch pool-retire --forget <email> on
the pool it leaves (a tombstone, so no machine pushes it back) and a write to
the one it joins. A pool may tier its accounts (tiers, email → primary |
secondary, default primary): since 2026-10-16 the reviewer pool holds
primaries that only the reviewers use and secondaries shared with the fleet's
pool on purpose, and the reviewers fall back to a secondary only when no
primary answers (dotclaude#270). propflow.staging, qa and sandbox left the
reviewer pool that day and stay in the fleet's; sandbox rejoined it as a
second secondary on 2026-10-22 (Gera). On 2026-10-26 those three accounts
were all capped for the week and every review died with no verdict; Gera:
"let's give it a total of five and two of them should be GitHub review
only." propflow.owner joined as the second primary and left the fleet's pool
(pool-retire --forget), and propflow.operator joined as a third secondary
and stays in both. On 2026-10-28 every one of those six was capped for the
week and reviews died in about 400ms with no verdict (propflowai#23290). So
propflow.agent and josegerag joined as secondaries (the fleet accounts with the
most weekly headroom, as measured 2026-10-28) and stay in the fleet's pool too.
The rotation ran at once rather than waiting its 2 hours.
The CI evals have their own seat (Gera, 2026-10-28, gera-propflow/dotclaude#344).
The CI evals read propflowai's CLAUDE_CODE_OAUTH_TOKEN / _FALLBACK, which used
to be the top two of the reviewer rank, so evals and reviews drained the same
two seats. A pool account can now carry the tier evals (propflow.test today).
Reviews reach it last, after every answering primary and secondary. The evals
reach it first; the rotation logs that order as eval rank (…) and publishes
the two eval secrets from it. It is not exclusive, because nightly.yml,
claude-type-sync.yml and the reviewers' legacy backstop read the same two
secrets. propflow.lab joined as a fourth secondary the same day.
It pages before reviews die (dotclaude#345). Each cycle, if the best
answering primary or secondary has under 15% of its weekly window left, it posts
🟠 to #alerts. If nothing answers at all, it posts 🔴. The two severities use
separate dedupe keys, so a same-day 🔴 is never folded into the earlier 🟠.
The near copy of both behaviours is rotate-reviewer-tokens.sh and its test
scripts/tests/reviewer-own-pool.sh (dotclaude).
The ccswitch row on /agents → Systems lists the reviewer
pool's accounts in their own table, read from the reviewer-tokens detail
below. A pool for Fede is the next one; nothing here assumes a fixed number of
pools.
The remote pool (AWS worker) (2026-10-28). Gera: our servers run on
dedicated subscription accounts of his, in their own pool, because they run a
subscription from AWS machines. That pool is claude-pool-remote (ccswitch
layout, one owner, never shared). josegarzadev@gmail.com joined first for the
worker. When the Lambdas and Vercel moved onto the pool the same day,
propflow.clara@gmail.com joined too, because one account is thin for all three.
Each left the fleet the same way: its live sessions were moved off with
ccswitch move (in place, nothing killed), then ccswitch forget on the
mini's bank and ccswitch pool-retire --forget on the fleet's pool.
Which secret the worker reads. ECS does not read the pool. The task
definition propflow-renewal-worker-prod maps ANTHROPIC_BACKGROUND_AUTH_TOKEN
to the AWS secret propflow-anthropic-background-auth-token, a plain
comma-separated bearer list. The remote-pool step below is the one path from
the pool to that secret. It publishes the pool's live bearers when they change,
and refuses any account the fleet's pool also holds. A running task keeps the
env it started with, so a change reaches the worker at its next deploy. That
secret funds background enrichment, test-property traffic and the Clara shadow
runs (subscription only since propflowai#23197).
The other two copies, same step. The Lambdas read SSM
/propflow/prod/ANTHROPIC_BACKGROUND_AUTH_TOKEN at cold start, so a publish
reaches them within minutes. Vercel production's ANTHROPIC_BACKGROUND_AUTH_TOKEN
is edited in place through the Vercel API and reaches the site at the next CI
deploy, never a manual redeploy. Both were hand-set until 2026-10-28. SSM held
Fede's three logins, three of Gera's fleet and reviewer seats, and three dead
ones. Vercel held Fede's three. The worker's secret had held the same three of
Fede's: clara@propflowai.co and two from his local pool.
That value was the secret's AWSPREVIOUS right after the 2026-10-28 publish;
the next publish rotates it out.
The mini only; off on the MacBook (2026-10-05, ADR-0050,
dotclaude#230). Gera: "we don't want the CC switch on Mac book, only on the
mini." On the Mac, ccswitch does not start, move or label sessions. pf
starts plain claude on the Keychain login, and the status bar's ccswitch
labels are mini-only. Every signal above describes the mini's files. The Mac
copies of those files go stale on purpose.
The walled sweep on time signal (2026-09-24, ccswitch#146, local-bin#256).
The wall rescue (walled_sweep: find a session parked on a usage wall, ping
it, then swap or move it) used to run only at the end of ccswitch snapshot,
after that job's network probes. launchd never overlaps ticks, so the rescue
arrived every 5–12 minutes, and the 3-day p90 time stuck after a wall was 86
minutes. It now has its own job, ccswitch walled-sweep (launchd
co.propflow.ccswitch-walled-sweep, every 30s). The job reads the snapshot
already on disk, with no build and no probe, and writes
~/.claude/ccswitch/walled.json (started_at, finished_at, duration_s,
snapshot_age_s, and the sweep's result). walled sweep on time is that
file's age. At 180s it goes Stale, and 180s is also the threshold the snapshot
job uses to decide whether to stand its own sweep down, so the page and the
tool agree on whether the fast rescue is alive. Nothing is configured anywhere: while
walled.json shows a sweep that finished ok less than 180s ago, the snapshot
copies it into sections.walled and skips its own sweep. Otherwise it sweeps
inline exactly as before, under the same flock (walled.lock), so the two
never sweep at once. The plist ships unloaded, so this signal reads Unknown
(no file yet) on a box until someone runs the manual launchctl bootstrap
(recipe in the plist header). That is correct, because the fast rescue is not
running there yet, even though the slow inline one still is. There is
deliberately no launchd signal for the job: a launchd signal reads
down when the job is not loaded, and a down signal marks the whole
ccswitch row down, while the file's age already says whether the job is
working. One gap to know about: a tick that fails (no readable snapshot)
still rewrites the file, with ok: false and a reason. That keeps this
signal fresh, but the snapshot job then sweeps inline, and it is account snapshot fresh that goes Stale. Measured on the mini, the sweep's read path fell
from 22–26s to about 1s (one batched ps eww in place of one per pane), and it
reaches no network unless it is actually moving a session.
The subagent-hold guard consulted signal, and why it is shaped this way
(2026-09-10, ccswitch#86). Every respawn path runs tmux respawn-pane -k, and
-k kills the pane's whole process group — so a switch or a wall heal takes
the session's Task-tool subagents with it. Measured that day: session
ab6d89b8 (tmux claude-arch) was respawned three times in under two hours
and lost five Workers mid-task, two of them twice. Gera: "whenever we do CC
Switch, it kills the sub-agents." A respawn whose session has live subagents
is now HELD — re-decided every tick, released at zero, and given way to after
an hour so a wall can still heal.
⚠️ The signal watches whether the guard is REACHED, not whether it fires.
ADR-0004's inert-mechanism clause is explicit that tests prove the rule and
cannot prove the rule is ever reached, and a guard that quietly stopped being
consulted would look exactly like a fleet whose panes never have subagents —
indistinguishable from working. So subagent_hold stamps
subagent-holds/.consulted every time it reaches a VERDICT, held or allowed,
and the fail-soft bailouts deliberately do not. A hold itself is an event, not
a heartbeat: holds are rare and should be, so a signal keyed on one would sit
red in the healthy case and train everyone to ignore it. The two-day window is
the same generosity rotation telemetry gets, for the same reason — a quiet
weekend fleet takes no respawn decisions and is not broken. Held sessions
themselves render from the detail block, which now carries
sections.subagentHolds alongside the account rows.
A lane that hit the wall stops vouching for the pane that needs rescuing
(2026-09-13, ccswitch#105). live_subagents() dates a lane by its transcript's
mtime, and a 429 wall leaves a fresh file on a dead lane. Measured 2026-09-13
02:58Z on the mini, session 266081f7: the hold read ten live lanes, and
all ten carried the wall as their OWN last record (isApiErrorMessage: true,
apiErrorStatus: 429, "You've hit your weekly limit"), aged 846–955s —
comfortably inside the freshness window, and every one already dead, because a
429 terminates a lane with no retry. On that reading the respawn that session
needed would have been refused: a session held out of its own rescue by ten
casualties of the very wall it needed healing from. A lane whose own last
word is a 429 is now retired from the list. It reuses walled_detected()
rather than re-deciding what "walled" means; it is never given the session's
drill id, so a drill planted for the SESSION cannot put words in a lane's
mouth; and it fails soft — an unreadable lane has no opinion and keeps whatever
the mtime rule said, never the other way round. Only a 429: a 500 or a 529 is
retried by the harness, so such a lane may be about to speak again. This is
worth the most on the respawn path, which kills every lane; a swap harms none.
⚠️ A lane has no process of its own, so process-based lane liveness cannot be
built — recorded in PITFALLS.md §6 Do NOT build, so the next reader does
not spend another night on it. Measured 2026-09-13: a lane's tool call is a
direct child of the PARENT with nothing in between, its environment names the
parent's CLAUDE_CODE_SESSION_ID and carries no agentId at all, and session
266081f7's ten concurrent lanes ran under one pid — sampled every 2s for
16s its child count was a constant 3, two MCP servers and one long gh poll
loop, so the count tracks TOOL CALLS, not lanes. Neither agent-<id>.jsonl nor
agent-<id>.meta.json records a pid. The coarse fallback — does this session
have a non-MCP tool child — is worse than useless: the one observed was an
until MERGED poll loop, which would have held its session's respawn
indefinitely.
The swappable sessions being born signal. ccswitch exec gives every new
session its own CLAUDE_CONFIG_DIR under ~/.claude/ccswitch/session-cfg/
with a .credentials.json in it, so that directory's mtime advances every time
the new birth path is exercised — the same anti-herd shape as session registrations flowing, one level earlier. Deleting swap_sessions (below)
removed the one thing that made the birth path's state legible at a glance, and
a birth path that quietly stopped producing config dirs would look exactly like
a fleet that spawned nothing. 24 dirs existed at 2026-09-13 03:36Z, the newest
still being written.
The respawn cluster is DEPRECATED, and the gate that retires it is
CHECKABLE (ccswitch#103). respawn_losses_capture, respawn_handoff_note,
respawn_handoff_deliver, the restart_notice re-arm injection,
move_session's respawn branch and the blanket subagent hold above are marked
THE LEGACY RESPAWN PATH — DEPRECATED 2026-09-12, STILL LOAD-BEARING: a
pre-cutover session is env-pinned for life, and a respawn is the only thing
that can rescue it. The cluster may be deleted when
ccswitch doctor --pinned-remaining
reads pinned 0 · unclassified 0 on BOTH machines, on a normal working day
with the fleet up. The counts are DERIVED per run — pinned_sessions_live()
reads each live Claude Code process's OWN environment out of ps eww and
applies the same three conditions swap_target() does — and unclassified
counts AGAINST the sunset, because a session that cannot be proven swappable
still has only a respawn. Exit 1 while pinned remain, exit 2 when the
answer cannot be derived, never a quiet zero (ADR-0016's rule again).
⚠️ The gate cannot be answered on the Mac today, and the reason is the last
pitfall in the section below. Measured 2026-09-13 03:36Z: the mini reads
pinned 27 · unclassified 0 · swappable 7, exit 1, on artifact
ccswitch-7b9d053efad4. The Mac reports ccswitch-deploy: CURRENT — against a
repo checkout 6 commits behind origin/main, at #101 — and its deployed
copy contains the string pinned-remaining zero times, so it ignores the
flag, prints ordinary doctor output and exits 0. A both-machines gate whose
unimplemented machine answers zero is worse than one that refuses: CURRENT
means the copy equals THIS machine's checkout, never that the checkout equals
origin/main.
⚠️ Why the gate is not a signals entry, and what would make it one. The
catalog is read-only by construction: relay/src/systems/catalog.ts evaluates
four signal types (file-age, log-match-age, launchd, self) and never
runs a command — the same safety property that keeps the per-account probes off
this page's poll interval. Nothing ccswitch writes carries these counts:
snapshot.json has no pinned field, and rotation.log's lines are JSON
objects while log-match-age needs a leading YYYY-MM-DD HH:MM:SS, which is
why rotation telemetry is a file-age in the first place. A sections.pinned
block in snapshot.json, or a stamp file on the subagent-holds/.consulted
model, would make it a signal. Until one exists it is a command a human runs,
like fleet-e2e.
What it is. The fleet's credential reflex. Every account is a 1-year
setup token minted by ccswitch mint <email> and shared through the pool
(bearer-only since 2026-09-03 — the browser-login path, its 10-minute
refresher and the keychain bank were deleted, not switched off; see
PITFALLS.md §8). The pool holds 8 accounts as of 2026-09-13 03:36Z;
propflow.clara@gmail.com joined on 2026-09-12 and is the picker's top choice
at 4% of its week — the only seat in the bank under 90% weekly at that
reading. A session that carries its bearer in its ENVIRONMENT binds it at exec
and is moved between accounts only by being respawned. The pin is a property of
that path and not of Claude Code, which re-reads its credential FILE per
request; see
ADR-0024.
Since ccswitch#103 that path is no longer how a session is BORN — merged
and deployed 2026-09-12, ccswitch exec births a swappable session
unconditionally, and there is no config key to turn it off — so the env-pinned
fleet is now a shrinking population of pre-cutover sessions rather than "the
fleet". This entry describes the rules that
survive as of Round 12 (2026-09-07), each one bought by a measured incident.
What the two signals and the detail block read. ccswitch snapshot
(launchd co.propflow.ccswitch-snapshot, ticking every 60s — a fast
re-probe of the 5h axis for accounts with live sessions, and a full build
every 600s) writes everything the tool knows: every account with live
5h/weekly/Fable usage, pool state, who is on it by session name, and the one
remedy; every token file with its probe result; an activity feed of every
decision with its numbers; sections.snapshot (fast/full timestamps, which
accounts were probed) and sections.walled (sessions found at a usage wall).
The Systems tab opens the row into that file. The relay only READS it: a
probe is one API request per account, and a sweep hung off a page's poll is
what rate-limited four accounts on 2026-08-26. Five missed ticks (300s) and
the row goes Stale; the signal and the detail block name the same file and
the same 300s on purpose and must move together. The session registrations flowing signal is the anti-herd canary (ADR-0004's inert-mechanism clause):
every live session renews its stamp every 10 minutes via temp-file + rename,
so the directory mtime advances with the flow — an hour of silence while
sessions run means the writer went inert again.
How a session gets and keeps an account.
- Bound at exec, once — because the bearer goes in the environment.
ccswitch execpicks the token before the process exists (#45); rotation never reaches into a running session. Read that as a fact about the env path, NOT about the CLI: a process with noCLAUDE_CODE_OAUTH_TOKENin its environment reads$CLAUDE_CONFIG_DIR/.credentials.jsonand can be moved while running — measured 2026-09-12, PID unchanged across the swap with 23 post-swap requests from a live subagent (ADR-0024). The spawn gate offers one file per ACCOUNT, ranked by 5h utilisation plus 8% per live session already on it (session_reserve), so leavers fan out instead of dogpiling the emptiest account; files the fresh snapshot saw capped sort last, unnamed copies last of all (#60, #61, #68). - Two kinds of session, and the kind is DERIVED not remembered. Since
ccswitch#99 a session is either env-pinned — a bearer in its
environment, which is every session born before the 2026-09-12 cutover — or
swappable:
no bearer in its environment, its own
CLAUDE_CONFIG_DIR, and a.credentials.jsonin it.swap_target()answers the question fromps ewwof the live process every time it is asked, because a stale "yes" kills a session that could have been swapped and a stale "no" is the five-Workers incident again. What the kind decides is the remedy, not the trigger: the switch fires on the same lines for both (§ the rule above), then a pinned session is respawned (the deprecated path, byte for byte, blanket subagent hold included) and a swappable one has its credential file rewritten in place — no kill, no lane loss. The swap is gated byswap_bar: it refuses a destination thatpreflightgrades 2, and with live subagents it demands a clean verdict rather than a marginal one, because a subagent carries no retry (one died 7.7s after its credential went bad, while a clean swap carried another through 23 requests). Live, and the only way a session is born (ccswitch#103, merged and deployed 2026-09-12).swap_sessionsis gone — the key, its one reader, and the docs that promised it — because a key made the fleet's shape a value on disk on one machine, and flipping it once was a fleet-wide change with a one-word commit. The only escape hatch is--no-swappable, a flag a HUMAN types for a test or an emergency, and it is loud on stderr and inrotation.logwhen it re-pins a session that has a credential dir. Measured on the mini 2026-09-13 03:36Z:pinned 27 · unclassified 0 · swappable 7, with 18 completed in-place swaps inrotation.logsince the cutover — e.g. session01ad649fat 2026-09-12 22:26 CDT, "swapped IN PLACE — pid 63987 keeps running, subagents keep their lanes … 4 live subagent(s) ride along, unharmed". The fleet converts by TURNOVER and by nothing else: nothing can move a running process off the env path, so the pinned count falls only as those sessions end. The sunset gate for the respawn cluster is above. ⚠️ccswitch holddoes not hold a swap. It is read at exactly one call site, insidehard_restart_now(), so it gates the RESPAWN — and a swappable session is still swapped in place under it. A fix is in flight as of 2026-09-13; until it lands, read a hold as protection against the kill, not against the credential rewrite. - A respawn acts on the pane that HOSTS the session, never on a stranger's
(#91). 2026-09-10 20:57:47Z: session
775cc26b— ansdk-clisession born twelve seconds earlier whose first prompt wasls— crossed a switch line and was respawned, and in that same second the fleet's Driver9e9f7a53and all eleven of its subagent lanes wrote their last line and stopped forever. Claude Code exportsTMUX_PANEinto every tool subprocess and every descendant inherits it, so a session started INSIDE another session's tool call read the HOST's pane out of its own environment, andrespawn-pane -kkills that pane's whole process group. The hold did not fail; it was asked about the wrong session, andsubagents: 0on that line is a true answer about the probe. The@claude_session_idstamp cannot arbitrate it either — a SessionStart hook writes it from inside the pane, so the probe re-stamps the pane with its own id on the way in, which is how the earlierclaude-archincident was misread as "the respawn resumed the wrong sid". The occupant is now read from the process table (pane_session_occupant()walks DOWN from the pane's own pid; the first claude it meets is the occupant, and a probe sits three levels below), checked athard_restart_nowand atwedge_heal, which respawns with-kwithout passing through it. ⚠️ Only a positive mismatch refuses — an unresolvable pane stays permissive, because refusing on unknown would strand exactly the sessions the wall healer exists to rescue (ADR-0016, one layer down: could not read the evidence and read it and said no must not share an exit path). - The 80% rule, and it interrupts. Every turn boundary, a session judges
its OWN account — never just the active symlink's — on a snapshot at most a
minute old. Any axis at its line → the session moves now, busy or not.
For an env-pinned session — a pre-cutover one, counted in the bullet above —
moving means being respawned through the spawn gate onto a different account
with a primed "continue" (#68); for a swappable one it means its credential
file is rewritten under it (same bullet). The TRIGGER is the same either way;
only the remedy differs. All destinations at or past the line → hold the least-bad
and say why. A restart already in flight short-circuits every other watcher
before it binds or logs, so one move is one
switchline (#70). - Fable's line is 0.95, not 0.80, and that is the opposite intent (#71). The other axes move a session BEFORE it is walled; Fable moves it as late as responsibly possible, because Fable is the best model and leaving it early is a loss, not a save. So a Fable-only move must PRESERVE: the destination has to have measured Fable room, and when nothing does the honest answer is to stay and say so — moving would trade a nearly-spent Fable window for another one. Unmeasured is not room.
- A destination has to be able to run THIS session's model (#74). Measured 2026-09-08: a session running Fable was evacuated on 5h into an account at 101% of its Fable window — logged in its own words, "moving to gera@propflowai.co (worst axis 101%)" — and walled 80 seconds later. The seat could take a turn; it could not take that session's turn. The session's model now orders the usable seats. ⚠️ A preference, never a veto: over its line Fable means the account cannot serve the BEST model (a downgrade), while 5h/weekly mean it cannot serve ANY turn (an outage). A hard filter would pin a Fable session to an account at 99% of its 5h window whenever the bank is all-Fable-hot, trading the second for the first.
- The restart cap is a runaway guard, not a ration (#75). It was three per
day, and on 2026-09-08 that parked three live sessions on an account at ~100%
Fable while
test@sat at 7% / 4% / 7% — a seat the picker was choosing correctly. The cap counted RESTARTS while its own rule claimed to count FAILURES ("a session that needs a fourth respawn has a problem respawning cannot fix"), and every restart it had counted was caused by a bug in the destination picker, not by the session. It now counts consecutive rapid restarts — an attempt older thanhard_restart_loop_window_s(300s) reads as zero, because a rescue that bought that much working time demonstrably worked — with the ceiling at 12: try every account twice, then stop. Gera's rule: "the goal is to get it working ... the counter is just there so we don't hit like a thousand." - A drill spends its own budget (#73).
switch-drillis refused when the production budget is gone — you do not rehearse on a session whose real rescues are spent — but it no longer SPENDS it. Session 70690853 escaped the Fable wall by running a drill on itself and was charged a production restart for it; two real rescues later it was stranded by the cost of its own escape. - The wall is a transcript event, healed like a wedge. Claude Code writes
"Usage limit reached · continuing automatically at …" into the transcript
the instant it happens.
autoContinueAtUsageLimit: false(dotclaude, both machines) makes the turn END there instead of parking on the SAME account until reset — the native wait was the exact state that never yields the Stop a respawn needs.ccswitch auto(hooks:UserPromptSubmit,Stop,SubagentStop,StopFailure) reads its own transcript tail and respawns onto the least-bad USABLE account (anything not itself at the wall); the 60s snapshot sweep does the same for a session whose hooks never fire. Feed events:walled-heal,hard-restart(#68). - Identity is a fact. Every token file resolves to an email through the
sha12 mint index; a file no index names is
unnamedand its sessions never register (#58, #60). A pane's SESSION id comes from the claude process's CHILDREN's environment (CLAUDE_CODE_SESSION_IDon the MCP servers and tool subprocesses) — the claude process itself carries none, and argv holds an id only for resumed sessions (#69). - Death is a server verdict. Only a 401 retires a bearer; the sha12
tombstone under the pool's
retiredmakes it stick (#58). The one remedy every label names isccswitch mint <email>— never/login, which rotates the refresh token under every session on the account. ⚠️mintitself was broken four ways until #107 (2026-09-12), and each one turns a correct paste into an unexplained failure:tty.setrawmeans only a CR submits, so the pasted code was never actually submitted; a failure invented a cause instead of keeping the CLI's own transcript; the authorize-URL regex swallowed the OSC-8 hyperlink's BEL; and a token could be filed with no index entry, which is theunnamedstate above.
"Which account am I on?" — ccswitch whoami (2026-09-11, ccswitch#94).
Reads the CLAUDE PROCESS's own bearer and names the account, in milliseconds:
no probes, no AWS, so a session that suspects it is about to wall never waits
on the bank. It exits 1 and says so — cannot determine, or not a session when there is no claude ancestor — rather than guessing; a bearer
reading that is right most of the time is worse than a refusal, because it
gets trusted. ⚠️ A subagent spends its PARENT's bearer, so whoami from a
lane reports the parent's account, and a fan-out CONCENTRATES rate-limit risk
instead of spreading it: fifteen lanes on one account wall in the same
instant. ccswitch switch-me is the move procedure, and since #108 it answers the
session that asked, per kind — swap_target() derives which from the asking
process's own environment, every time. The pinned branch keeps its wording
verbatim (which wall you actually hit — a Fable wall needs /model, not an
account move; what a respawn costs, namely the pane's whole process group; what
to land first; and why spawning a fresh session beside the loaded one usually
beats moving it). The swappable branch is a reference, not just a
correction: what to run (ccswitch switch-drill --sid <me> --self); that
there is no --to — no verb takes a destination for a loaded session,
best/list only show the picker's view and checkout binds nothing; what
will refuse you and why a refusal is the tool working; that a walled session
should say so in one line rather than go quiet, because a walled session is
alive and reads as dead from outside, a confusion that cost two Drivers about
an hour on 2026-09-12; and that with no live process a swap is the wrong tool
— that session needs ccswitch exec -- claude --resume <sid>, which binds a
bearer before the process exists. It exists because a helper session rescuing a
walled Driver that night could not learn the swap route from the tool, read
the source instead, and then did it right by hand with all ten lanes alive.
Long form: PITFALLS.md §1 and §2.
What still lies, and how to read it (the instrument caveats that cost real hours this week):
Bare
ccswitchand the statusline's account field show the FLEET DEFAULT, not your own account. Measured 2026-09-11: a coordinator with 15 running subagents was told it was at 98% of its session limit, read three surfaces, concluded 34% and fine, and argued the point twice. All three readings were correct and none was about the caller — bareccswitchranks the BANK and puts its arrow on~/.zprofile-token's account;ccswitch leaseswas EMPTY, which is the normal state of a healthy fleet and reads as "nothing is wrong"; the statusline names the right account but AS A NUMERAL, decoded by runningccswitch— i.e. by landing on the fleet default. The34%and101%it reported weregera@propflowai.co's 5h and Fable rows, verbatim; the session was onpropflow.evals@at 8% of its week. Forty minutes later the account hit 100%: seven subagents died mid-task on 429s and the rest died with the session. ⚠️ The capability existed the whole time —session_account()resolved it from 2026-09-01 and was called at exactly one site — and no command printed it. A built capability with no surface is indistinguishable from an absent one. Fixed in #94:whoami, andlist/doctor/leases/preflightrelabelled so the default can no longer be read as the caller."new sessions rotate" means NEW ones. The statusline says that when the account a session is bound to has crossed a line, and it is telling the exact truth: the spawn gate will send new sessions elsewhere, while this one moves only at its next turn boundary or on the 60s sweep — §1, an env-carried process cannot change its token, so for a PRE-CUTOVER session the only move is a respawn (that limit is the env path, not the CLI — ADR-0024). Asked on 2026-09-08 why a session reading
⚡ ACCT s38% w99% — new sessions rotatehad not rotated, the answer was that it had been evaluated correctly and was blocked further down (the restart cap). And a session not launched throughccswitch exec— an interactiveclaudeon the Mac, say — never rotates at all: no fleet bearer is bound to it, so nothing in ccswitch can move it, even though it spends the same per-account quota the fleet is competing for.EVERY DETECTOR HERE IS A CAUSE-DETECTOR; THERE IS NO EFFECT-DETECTOR. Each thing on this page watches for a specific known failure. Nothing watches for the outcome those failures produce — a session sitting on an over-line account while a usable seat exists. That is why 2026-09-08 unfolded as four separate discoveries in one night, each hidden by the one before it (the blind Fable axis masked the bad destinations; the bad destinations masked the counter), and why a third stuck session was found only because it moved in the same sweep as the two being investigated. One assertion over the outcome would have caught all four at once. It does not exist yet.
A registration is not a process. The snapshot counts sessions by their reg stamps; a walled session had rotated its registration off morpheus@ while its process never moved (2026-09-06). The process table is the truth —
sessionNamessay who claims an account,pssays who runs on it.The meter can read over 100%. Anthropic's own 5h header reports the ratio of spend in flight, so 108% is a soft overage, not a wall we punched through. The 80% line is a trigger; with a 60s meter it moves a session ~10 points before the wall, but it is not a ceiling and nothing should claim "we cannot hit a limit" — claim "no session stays walled".
A fresh session with no child process yet has no readable id (no MCP servers configured, no tool call made): the drill, the sweep and the respawn cannot find it. Every mini session carries two MCP children from boot, so this is a documented blind spot, not a live one.
The deployed copy must equal
origin/mainin BOTH directions. On 2026-09-03 it was 142 lines AHEAD (a hand-deployed feature in no commit) and onlydiffagainst main said so (#64). Copying the file is the deploy; the launchd plists in local-bin carry a__HOME__placeholder and install through the README recipe (sed "s|__HOME__|$HOME|g") — a rawcpleft the 60s timer pointing at a path that does not exist (2026-09-07). ⚠️ AndCURRENTis a narrower claim than it reads as — it means the copy equals THIS machine's checkout. See the sunset gate above, where the Mac wasCURRENTon a checkout 6 commits behindorigin/mainand so could not answer the gate at all.
Proving it works — on demand, never by waiting for a cap.
ccswitch switch-drill --sid <sid> / walled-drill --sid <sid> plant a
synthetic reading for one session and run the REAL path (every event stamped
drill: true); fleet-e2e --switch / --walled spawn a throwaway session,
drill it, and pass only if it comes back on a DIFFERENT account, takes a
second real turn there, re-registers, and the feed carries the drill-stamped
switch (or walled-heal) → hard-restart: restarted. Four consecutive
passes on 2026-09-07.
⚠️ A drill names the remedy that actually fired (#108). On a swap it used
to print "the pane respawns in ~2s … watch for hard-restart in
rotation.log"; measured on a real session the entry read "how": "swap", pid
99958 unchanged, and no hard-restart line was ever written — the
Driver was sent to watch for a signal that could not appear. The armed line
now reads the remedy off the rotation entries the drill itself wrote, and names
the signal that will really show up. Same fix on walled-drill. And the swap
debounce stamp moved out of the check it sat inside: _switch_due stamped as
it READ, before the destination probe and before the write, so being ASKED
counted as a swap HAPPENING — a birth counted as a swap, and every attempt
for the next 120s was refused with "swapped within the last 120s" for a swap
that had never occurred. fleet-e2e alone is PITFALLS §0 for any credential
change; a run costs a real session start, so never on a timer.
Knobs (ccswitch config <k> <v>; defaults in DEFAULTS): switch_at
0.80 · switch_at_axes {"fable": 0.95} · session_reserve 0.08 ·
hard_restart_max_per_day 12 · hard_restart_loop_window_s 300 ·
snapshot_full_every_s 600 · auto_walled_heal true.
⚠️ swap_sessions was here and is deleted, not defaulted false (#103); the
name exists in no code and no string. Swap-in-place is the only birth path, so
there is nothing to configure — see the bullet above rather than looking for
the knob.
⚠️ The DEFAULTS dict is built at import, before those constants exist, so it
carries literals — a drift guard in tests/wedge-heal.py asserts the pair
agrees. That pair silently disagreeing is how the cap stayed at 3 while the
constant read 12, during the change that raised it.
⚠️ snapshot_full_every_s and auto_walled_heal are constant fallbacks, not
DEFAULTS entries — read as cfg.get(key, CONSTANT). The effective values
are as listed (verified against the deployed tool, not the source), but
ccswitch config will not print them until one is set, so a blank there is not
the same as an absent knob.
Where the history lives. PITFALLS.md §8 in the repo is the dated
incident log (one row per defect, the fix beside it); the phase tracker
carries Rounds 8–12; ccswitch-never-walled is the write-up of why 108%
happened with the 80% rule live and what closed it.
Appendix — everything linked from this entry
agent specialists
| What it does | The named JOBS a Driver delegates to instead of doing itself — decider, nudger, cleaner, optimizer, finisher, updater, slacker, fixer. Seven are ACTIVITIES and exactly one (the fixer) is an AGENT, settled 2026-09-12 by Astra's test (bounded contract -> activity; investigative ownership -> agent) and ADR-0025 decision 8: a determination a Driver makes repeatedly has to be an activity with a script behind it or a Temporal workflow cannot replay it, because a model asked twice may answer twice. ALL EIGHT are catalogued as their own systems (decision-ladder, nudge, cleaner, optimizer, finisher, updater, slacker, red-main) and are NOT duplicated here; `updater` and `slacker` were the last two built, both on 2026-09-12, and both have their own entries below. |
|---|---|
| Kind | orchestration |
| Repo | https://github.com/PropFlow-Technologies/agentflow-relay |
| Deployed at | — |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → agent specialists |
Liveness signals
| Signal | How it is read |
|---|---|
| roster snapshot fresh | file-age · ~/.claude/jobs/specialists.json · stale after 1440 min |
Detail document: ~/.claude/jobs/specialists.json (the Systems tab opens this row into it).
The convention (Gera, 2026-09-12): <verb>er specialist. It names the
FUNCTION rather than a personality, which is the only form you can ask a
liveness question about — "is the updater alive?" has an answer, "is the
audit agent alive?" does not.
| Specialist | What it does | Housing | Catalogued as |
|---|---|---|---|
| decider | asks Fable and Astra INDEPENDENTLY on one snapshot (ADR-0019), returns a ruling or NEEDS HUMAN |
activity | decision-ladder — active, has a canary. Answers 61% without a human against a ~95% target |
| nudger | pokes an idle session; carries no claim about its state | activity | nudge — active |
| updater | writes the closing prs ref onto the action item so status can derive — it RECORDS the sub-agent's determination, it does not judge full-vs-partial closure |
activity | updater — built 2026-09-12, its own entry below. Rung 4, historically ownerless; housed as an agent until 2026-09-12, when Gera removed the judgment it was housed on |
| slacker | posts when an action item is observably done — per CLOSED ROW, never per PR, as the Agent Smith bot | activity | slacker (slack-poster) — built 2026-09-12, its own entry below. Runs every 600 s from launchd co.propflow.slack-poster and sweeps the published dock each tick |
| cleaner | tears down sensors, processes and worktrees a closed task left | activity | cleaner — hourly via co.propflow.worker-cleaner (tasks, the ADR-0037 worktree/cache phase, then cleaner-jobs). The co.propflow.pr-cleaner ORPHANED-PR job was retired 2026-10-10 (0 PRs closed in 305 runs). Close signals must trigger it |
| optimizer | says which rows a Driver may work AT ONCE, only where independence is PROVABLE | activity | optimizer — built and on PATH, ZERO automated callers. This is the Driver's parallel-vs-serial decision (ADR-0025 §3): wire it, do not rebuild it |
| finisher | recounts the published dock and exits 0 ONLY at 100% — the one thing allowed to say the work is done, and the one specialist that may never be switched off at setup | activity | finisher — active, wired to the Stop hook at drive-gate.py:726; 64 runs on 2026-09-12, every one exit 1, correctly refusing a stop at 48% |
| fixer | takes a red main, fixes it and merges — scope is the RED, not the work that caused it |
agent | red-main — active; it appoints the fixer rather than being one. --allow-push was already passed at the mint (redmain.ts:1363); since 2026-09-12 the merge authority is stated too |
Which of them are ON, and what each one points at, is driver-setup — the
SETUP table a /driver run starts through (its own entry below). All eight ON
by default, opt-outs by number, finisher never off, and every parameter
resolved against a real referent at setup and re-checked on use. An enabled
specialist whose ref does not resolve — or that has no implementation at all —
refuses the start, which is the mechanical answer to the updater failure this
table records twice. It reads the box, not this file — builtness is resolved
at setup time, so the roster rows above stay the record of intent and the table
stays the record of fact.
One agent, seven activities (settled 2026-09-12; this line read "Two agents,
four scripts" and miscounted its own table). The fixer is the only member
that genuinely needs investigative judgment; the other seven are bounded
contracts and are Temporal ACTIVITIES with scripts behind them. The Driver is not
on this roster at all — it is the coordinator that calls it, handed a dock and
assigning its action items to sub-agents. Astra's test stands unchanged —
bounded contract → activity, investigative ownership → agent — and applying it to
the updater after Gera removed its judgment is what moved that one across.
That is the point, not an accident of scale.
⚠️ The fixer announces, and it announces by WRITING, not by messaging. ONE
message when it takes a red, ONE when main is green, so no other lane tackles it
— as a write to the shared surface (~/.claude/jobs/red-main/claim.json +
status.json), never N messages. A fixer that messages every agent IS a caller
(ADR-0003) and cannot
know who "everybody" is anyway.
Why the fixer is an agent and red-main is not it. red-main holds ONE
atomic claim naming the single fixing lane, so every other would-be fixer —
Smith's alert lane, a second session, a person — can read who owns it and stand
down. It appoints; it does not repair. The lane it names is the fixer, and
diagnosing a red build is the one job here no script can do: the failure is
different every time, and the first question is always whether the red is real
or infra (a reclaimed runner, an OOM) — a judgement that has been got wrong in
both directions on this fleet. So red-main answers who owns this, which is
mechanical and correctly a reflex, and the fixer answers what is wrong, which
is not.
There is deliberately no auditor. It was a workaround for the Driver grading
its own homework; make the done-check observable (pr:owner/repo#N merged is an
OBSERVED kind, file:/exit: are ATTESTED) and the job disappears. Its
derivable half belongs in sensors, which cannot be switched off. A decider
answers a question somebody put to it; an auditor asks questions nobody put, and
only the first is a job you can schedule.
The routing rule this roster exists to enforce — Gera, 2026-09-12: "it needs
to stop asking for decisions and use the decider specialist." Ask the decider
first; raise a block only on NEEDS HUMAN, or when the 2026-09-12 escalation
test says a human: is it IN THE PLAN, and is it REVERSIBLE? In-plan AND
reversible — the decider rules, including deleting or adjusting data when the
plan calls for it. Out of plan, OR irreversible — a human (ADR-0025 decision 7).
The older category list below is superseded as an independent gate; its entries
survive as INSTANCES of out-of-plan or irreversible, so an in-plan reversible
production write is now the decider's. The old wording: when a human is structurally
required (a production write, a customer promise, money, or a session changing
its own acceptance bar). Full reasoning and the worked examples are in
docs/the-specialists.md.
Appendix — everything linked from this entry
driver-setup (the table a /driver run starts through)
| What it does | The SETUP step of a `/driver` run: prints one row per specialist — `# | specialist | enabled | parameter-ref | validation` — with every specialist ON by default, opt-outs BY NUMBER, `finisher` never off, and the ONE parameter each specialist needs resolved against a REAL referent (a dock slug fetched from propflow-docs@main and parsed to rows with the finisher's own parser, a Slack channel conversations.info answers for, a repo GitHub answers for, a task dir, a readable directory, a ladder fable-decide really has a flag for). `driver-setup --start` is what "driver start" runs, and an enabled specialist that cannot work REFUSES it. Mid-run, `--check <name>` re-validates one row, raises an invalid ref with the decider and ALWAYS exits 0 — nothing in this flow waits for a human. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/driver-setup |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → driver-setup (the table a /driver run starts through) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/driver-setup.log · matches driver-setup · stale after 10080 min |
The failure it exists to prevent, which is on this record. updater was on
every mental checklist in this fleet for weeks and was never built. Nothing ever
wrote a closing prs ref onto a dock and nobody noticed, because "the updater
is on" was a REMEMBERED fact. Remembered facts go false silently. Every
parameter in the table is therefore a REFERENCE, resolved at setup and
re-checked on use, never free text.
It had nowhere to live. task.json has ten keys and none names a
specialist; "specialist" appears zero times in start-operator,
operator-here, canon/drive-gate.py, task-event or register-work. The
record is a sidecar at ~/.claude/jobs/tasks/<slug>/driver-setup.json, beside
acceptance.json and for the same reason — the Supervisor round-tripped
task.json through an allowlist that erased fields it did not know (it is
deleted, ADR-0067; other writers can do the same, so the sidecar stays).
Two refusals, because they are two repairs.
| exit | what it means | what to fix |
|---|---|---|
| 5 | an enabled specialist has an INVALID ref, or no implementation at all | the ref, or opt it out by number |
| 6 | an enabled specialist's ref could NOT BE MEASURED (gh missing, Slack unreachable) |
the checker — nothing is known to be wrong |
Collapsing 6 into 5 sends somebody to repair something that is fine, which is this catalog's most repeated failure shape.
A valid ref on an unbuilt specialist is the SAME failure as a bad ref — a
live Slack channel in front of a slacker nobody wrote is still "on and
silently doing nothing" — so builtness is checked as a separate fact and blocks
start too. Which specialists that catches is resolved against the box at setup
time, never copied from this page: a catalog row is a claim about the past and
the whole point of the check is that such claims go false.
--start also writes the loop's FINISH marker from the validated
finisher slug. drive-gate.py:726 reads that file to refuse a Stop while the
board is short, and nothing on this machine ever wrote it — the one
specialist that was wired was wired to a marker nobody produced.
finisher can never be switched off, by flag or by editing the record: a
record hand-edited to enabled: false is coerced back ON at load, because the
file is the easier thing to change than the flag.
Arming cannot route around it. operator-here refuses to arm (exit 8) while
a task dir carries a driver-setup.json with no startedAt
(gera-propflow/dotclaude#55). Opt-in by construction: the record exists only
where driver-setup wrote one, so every other session is untouched.
Pinned by tests/driver-setup-an-unresolvable-ref-cannot-start.py — a real
404ing dock slug against real GitHub, in an isolated jobs root, with five named
sabotages each proven to turn the suite red.
Appendix — everything linked from this entry
cleaner (retire what finished — tasks, worktrees, and dead hosts' lock files)
| What it does | Archives task rows that reached a terminal state, so the things that walk the fleet stay cheap. Work that scales with item count on a fixed cadence, with nothing retiring the dead ones, is how /api/systems reached 30s and how the AppFolio flood was shaped; this is the retirement half. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/cleaner (launchd co.propflow.worker-cleaner, hourly apply: tasks, then caches/worktrees, then cleaner-jobs) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → cleaner (retire what finished — tasks, worktrees, and dead hosts' lock files) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches worker-cleaner exit= · stale after 1440 min |
| hourly job loaded | launchd · co.propflow.worker-cleaner |
| worktree GC judging | log-match-age · ~/.claude/jobs/specialists.log · matches cleaner-worktrees mode= · stale after 180 min |
--mode has no default and that is the design. A default of dry-run is
what let a previous backfill in this fleet be "run" for a year without writing
anything — a default is what makes it ran and it printed a plan the same
observation. Terminal states are an allowlist, never status != open: a
status nobody has seen before must not be swept because it failed a negative
test.
Appendix — everything linked from this entry
test-inbox (read test@propflowai.co, read-only)
| What it does | Lets a session confirm that synthetic test mail actually ARRIVED at test@propflowai.co, the inbox every test+<tag>@ address lands in since propflowai #17939 moved test mail off the Slack-mirrored public hello@. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | ~/.local/bin/test-inbox (token ~/.config/test-inbox/token.json, gmail.readonly, test@ only) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → test-inbox (read test@propflowai.co, read-only) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches test-inbox exit= · stale after 20160 min |
Appendix
- https://github.com/PropFlow-Technologies/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
cleaner-jobs (lock files of dead hosts)
| What it does | Hourly, from an allowlist only: unlinks jobs/decision-wake/<sid>.lock and jobs/nudge/compacting-<sid>*.lock whose sid has no live host by fleet-medic host_verdict and that have been unused for more than 24 h (a decision-wake lock is never written, so its last use is the later of it and its <sid>.json) — only after taking their flock; a held lock is kept. UNKNOWN keeps and is counted. At most 200 removals a run. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/cleaner-jobs, run as the last phase of co.propflow.worker-cleaner's hourly apply (merged 2026-10-10, Gera 2026-10-10, fleet cleanup step 2; its own co.propflow.cleaner-jobs job is retired) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → cleaner-jobs (lock files of dead hosts) |
Liveness signals
| Signal | How it is read |
|---|---|
| hourly job loaded (worker-cleaner) | launchd · co.propflow.worker-cleaner |
| sweep judging | log-match-age · ~/.claude/jobs/specialists.log · matches cleaner-jobs mode=apply · stale after 180 min |
--mode has no default (the cleaner's rule), a repeated flag is a usage
error (a plan must never become an apply because a later argument won), locks
are unlinked only after this process takes their flock, and every removal is a
cleaner-jobs REMOVED kind=… sid=… line beside the run line.
cleaner-jobs --mode=dry-run prints the plan with every UNKNOWN named.
Appendix — everything linked from this entry
wt-merged-notify (a merged PR's session takes its worktree down)
| What it does | Every 2 minutes, finds worktrees whose PR just merged or closed, works out which live Claude session worked in each (a process inside it, else the session whose recent transcript names it) and prompts that session through msg: take it down now with `wt-done`, no permission needed. wt-done deletes through worker-cleaner's paced, stall-probed _rm. The per-session fast path of ADR-0037; worker-cleaner stays the machine-level backstop. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/wt-merged-notify (launchd co.propflow.wt-merged-notify, every 120 s) + ~/.local/bin/wt-done |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → wt-merged-notify (a merged PR's session takes its worktree down) |
Liveness signals
| Signal | How it is read |
|---|---|
| tick scheduled | launchd · co.propflow.wt-merged-notify |
| judging | log-match-age · ~/.claude/jobs/wt-merged/notify.log · matches tick considered= · stale after 15 min |
Why it exists (2026-10-01). The mini's data volume hit 0 GB free on
2026-09-29 and again on 2026-09-30. 491 of 688 propflowai worktrees belonged
to PRs already merged or closed: each task got a worktree, the PR merged, and
nothing removed it. Gera: "whenever we merge something it should prompt [the
session] ... it's been merged, go ahead and take down the worktree, no
permission is needed ... so I don't get asked." Decision record: ADR-0037,
amendment of 2026-10-01.
How it works. One gh pr list --state closed per repo under
~/code/PropFlow that has linked worktrees; PRs closed in the last 3 days are
matched to worktrees by head branch. Each worktree is handled once
(~/.claude/jobs/wt-merged/notified.json). The first run records a baseline and
sends nothing, so an existing backlog stays the cleaner's; at most 3 prompts go
out per tick. A worktree with no live owner is logged NO-OWNER and left to
worker-cleaner. It never deletes anything itself. wt-done refuses if the
caller stands inside the worktree, a session was launched there, another
process works there, or the PR is not finished; it archives leftovers to
~/.claude/jobs/host/wt-archive/, never deletes the branch, and removes the
worktree only when worker-cleaner's _cheap_to_remove says so (linked
.pf-shared install, or no node_modules). An older full-clone worktree gets
its build output cleared and waits for the cleaner's nightly window.
wt-merged-notify status lists what was sent.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
optimizer (provable parallelism)
| What it does | Reads the dock and says which rows a Driver may work AT ONCE — only where independence can be PROVEN from the rows' own declared paths. Exists so a Driver's sub-agent fan-out comes from evidence instead of from a guess about overlap; ADR-0025 decision 3 routes the Driver's parallel-vs-serial decision through this tool rather than rebuilding it. ZERO automated callers as of 2026-09-12. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/optimizer |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → optimizer (provable parallelism) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches work-optimizer exit= · stale after 1440 min |
"I could not find a conflict" is not independence. A pair is reported parallel-safe only when both rows name paths and those paths are disjoint; a row that names none is UNKNOWN, which is a third answer and not a pass. First honest run over 147 rows: 39 parallel, 27 conflicting, 81 unknown — and the 81 is the useful number, because an earlier cut reported 145 of 147 as naming no file by reading a hand-written list of fields instead of deriving them from the row.
Appendix — everything linked from this entry
finisher (the finish line)
| What it does | Recounts the PUBLISHED dock a task drives (resolved by dock-of from --task, or named with --slug; no default) and exits 0 only when every row reads merged or shipped. The only thing allowed to say the board is done; everything else describes it. Ships with dock-claimed-items-merged, the per-task gate two Driver task.json files and canon/operator-SKILL.md had named for weeks while no such command existed on PATH. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/finisher |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → finisher (the finish line) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches dock-finisher exit= · stale after 1440 min |
Gera, 2026-09-12, after a Driver reported its goal met while the dock read 130/285: "we only stop until we hit the completion 285 or whatever the number is... we shouldn't literally claim done until we run a script and it returns a hundred percent."
What it exists to prevent. The goal given was "the dock reads N of N done
on origin/main". It was rewritten to "every closing PR named on this Driver's
acceptance list is merged" — on the true observation that the original N (222)
miscounted prose as rows. But the fix for a wrong count is to RECOUNT. What
replaced it was a list the Driver writes for itself, which no input can make
come out false, so it scored 14/14 with every PR genuinely merged while the
board had not moved. finisher recounts. That is the whole job.
Why the drift was structural, not a Driver going rogue. An exit:<job>
acceptance item is attested, not observed: the evaluator runs no command and
looks for the check string verbatim in events.jsonl. pr:owner/repo#N merged
is observed against GitHub. So the board-pinned criterion was a
self-attestation about a command that did not exist, while the PR list was the
only kind anyone could verify — and both live acceptance lists drifted to PR
refs. The harness was paying for the wrong thing.
| exit | finisher |
dock-claimed-items-merged |
|---|---|---|
| 0 | every row merged or shipped | every claimed row merged or shipped |
| 1 | the board is SHORT (how many, by group) | some claimed rows are not done |
| 3 | — | no live claims: absent, empty, or all RELEASED |
| 4 | the dock could not be fetched, or parsed to zero rows | same |
| 5 | — | a claimed slug is not on the dock at all (renamed) |
3, 4 and 5 are deliberately neither 0 nor 1. "I could not measure" must not
read as "done", and must not read as "the board is red" either — a broken
gate reported as a red board sends somebody to repair rows that are fine. An
empty claim list exits 3 rather than passing vacuously, which is the failure
this whole roster keeps meeting: an absent input rendering as a positive
verdict. Pinned by tests/finisher-absence-is-not-done.sh, six cases, nearly
all negative.
It reads the published dock over the API at a ref, never the working tree. A finish line the runner can edit is not a finish line.
Appendix — everything linked from this entry
dock-of (which board a task drives)
| What it does | Resolves the ONE dock a task drives from its setup record (driver-setup.json's finish-line row) and refuses when none is recorded. Every board reader — finisher, dock-claimed-items-merged, driver-brief, slacker, fleet-grader, critic-poll, plan-review — resolves through it; there is no default board anywhere (ADR-0036). |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/dock-of |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → dock-of (which board a task drives) |
Liveness signals
| Signal | How it is read |
|---|---|
| a board reader resolved a dock recently | log-match-age · ~/.claude/jobs/specialists.log · matches dock-finisher exit= · stale after 1440 min |
Why it exists. On 2026-09-29 four Drivers ran four docks while seven tools
defaulted to portfolio-architecture-phases, the only dock when they were
written (2026-09-12). The homepage Driver's brief measured its claims against
the portfolio board and called its own rows "not on the published dock". None of
it said so, because a default never does.
Where the answer lives, and why not task.json. In the setup record, written
by driver-setup, and nowhere else. task.json was round-tripped through the
Supervisor's allowlist, which erased a field it did not know (deleted,
ADR-0067), and a second copy is a second fact to keep true (ADR-0036 decision 1). Other repos shell out to
dock-of; they do not re-read the file.
| exit | meaning |
|---|---|
| 0 | resolved (with --verify: and the finisher can count it) |
| 2 | no dock recorded — the line names what is missing |
| 4 | --verify: the finisher cannot read or count that dock |
dock-of --live lists every live task and its dock, or why it has none.
Appendix — everything linked from this entry
docs-heal (the shared docs checkout unjams itself)
| What it does | Runs at the start of every `bin/docs` add/rm/publish, before a byte is written, and heals the shared propflow-docs checkout when a git operation died half-way in it: stale locks, a stale rebase/merge/cherry-pick state, stranded dirty paths, local-only commits nobody pushed. Backs up to backup/docs-heal-<ts>, stashes stranded edits under a timestamped label, resyncs with origin. Never touches a checkout where a git process is live (lsof by cwd). |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflow-docs |
| Deployed at | ~/code/PropFlow/propflow-docs/bin/docs-heal |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → docs-heal (the shared docs checkout unjams itself) |
Liveness signals
| Signal | How it is read |
|---|---|
| a publish ran its preflight and the health check recently | log-match-age · ~/.claude/jobs/docs-heal/health.log · matches docs-publish: · stale after 1440 min |
Why it exists. The shared checkout jammed three times on 2026-09-29, each unjammed by hand, and every decision raised meanwhile sat publish_pending on the mini only. The incident shapes, the thresholds, the exit codes and the controls that reproduce each jam live with the code (the near copy is the bin/docs-heal docblock and propflow-docs#262) and are not restated here, so this entry cannot drift from them.
Appendix
- https://github.com/PropFlow-Technologies/propflow-docs#readme — the README (no docs declared in SYSTEMS.md yet; add some)
docs-publish-health (is docs publishing actually reaching origin)
| What it does | An alarm that can come back BAD. It checks for two things: local-only commits in the shared docs checkout, and open decisions that are publish_pending and missing from their page on origin/main, each past a threshold held in the docs-publish-health docblock and local-bin#305. Posts to #agent-smith via the Smith bot when BAD, debounced, with one post on recovery. fleet-grader's `docs-publish` row reads it. Read-only; never fetches. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/docs-publish-health |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → docs-publish-health (is docs publishing actually reaching origin) |
Liveness signals
| Signal | How it is read |
|---|---|
| docs publishing read OK recently | log-match-age · ~/.claude/jobs/docs-heal/health.log · matches docs-publish: OK · stale after 1440 min |
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
updater (the closing ref, so status can derive)
| What it does | Writes a merged PR's closing `prs` ref onto the board row it closes, in the form propflow-docs/bin/refresh-tracker parses, so the row's status DERIVES. It writes the REF and never a status. Full-vs-partial is COMPUTED from the evidence kind — every check OBSERVED (pr/run/slack) and every pr: verified MERGED writes a bare string (full); any ATTESTED check (file/exit), any observed check that did not verify, or any kind outside the five writes {closes: partial, reason} and prints the `blocked raise` line. An unrecognised kind fails CLOSED. The two sets are a MIRROR of the canonical ones in agent-smith `src/agent_smith/operator_task.py:520` (read by `check_is_observable`), copied rather than imported because this runs from PATH in another repo — so that copy is the known divergence risk. The sub-agent's own `--closes partial` is never promoted by evidence: nothing-remains is a claim about the complement, which no check kind can observe, so upward movement stays `completed_by`'s job. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/updater |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → updater (the closing ref, so status can derive) |
Liveness signals
| Signal | How it is read |
|---|---|
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches updater exit= · stale after 1440 min |
What it exists to end. A row's status is derived from its own prs field, so
merging moves nothing: a merged PR that no row names reads open forever.
Measured 2026-09-12 — the portfolio-architecture dock held at 141/291 across
fourteen consecutive refresher commits while real work merged in propflowai.
The hop between "merged" and "the board knows" is one ref, and until now no
specialist owned it.
Housing: ACTIVITY, and it changed hands the same day. Astra housed it as an AGENT because full-vs-partial closure was a judgment call. Gera removed the judgment — "the updater should be just focused on passing information into the docs… it should be self-determined" — so under Astra's own test (bounded contract → activity; investigative ownership → agent) it is an activity. ADR-0025 decision 8: the test survived, the answer changed.
⚠️ THE WRITE PATH HAS A SECOND WRITER AND THIS ONE IS BUILT FOR IT.
propflow-docs/bin/refresh-trackers rewrites the same artifacts from a launchd
tick every 300s in a checkout every session shares, and on 2026-09-12 that race
destroyed the content of a merged PR — ten rows, a clean fast-forward, every
channel reporting success. So updater never touches a working tree: it
reads the artifact through the GitHub contents API, keeps the blob sha, and
PUTs pinned to it, so GitHub itself answers 409 if anything landed in between.
It re-reads and re-applies rather than forcing, and then verifies by reading
origin/main again — a write that reported success is not evidence it
survived. Pinned by tests/updater-a-moved-base-never-loses-a-row.py, ten
controls; the sharpest sabotage pins the PUT to a freshly-read sha instead of
the derived-from blob and reproduces the incident exactly (exit 0, no conflict,
the co-tenant's row gone).
Appendix — everything linked from this entry
slacker (announce a closed row, once)
| What it does | Posts to Slack when an action item is observably DONE — per CLOSED ROW, never per PR ('per PR is noise; per closed row is a thing worth reading'). It sweeps the PUBLISHED dock over the API and announces a row that MOVED into merged/shipped since the last observation, carrying the row's own title, its own prose and its closing PR links — never a paraphrase. It posts as the AGENT SMITH BOT, the automated-surface path in post-updates/SKILL.md, and never as Gera: that skill's MCP identity covers its own broadcasts and nothing else. Idempotency is a receipt CLAIMED with O_CREAT|O_EXCL BEFORE the post, keyed on the ROW and the STATE IT MOVED TO, so a retried sweep, a second copy on the box, or a run killed between the post and the state write all find the claim and post nothing; an unsettled claim is never re-taken automatically (exit 6, --reclaim). It never posts on a guess: a transition is required, the row must carry an OBSERVED closing ref (a bare string or closes:full — partial-only, which is what updater writes for attested file:/exit: evidence, is skipped), an absent prior observation is a baseline rather than a closure, and an unreadable dock or a zero-row parse is exit 4 with the state untouched. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/slacker → slack-poster; launchd co.propflow.slack-poster every 600 s (system domain) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → slacker (announce a closed row, once) |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.slack-poster |
| run recently | log-match-age · ~/.claude/jobs/specialists.log · matches slack-poster exit= · stale after 60 min |
Which poster, and the line that decides it. post-updates/SKILL.md splits
Slack by SURFACE: its own broadcasts go out as Gera through the MCP, with
"no bot-token fallback for update posts. Ever."; the automated surfaces —
"heartbeats, escalations, task-event mirrors, #agent-smith alerts/ops,
digests" — go through the Agent Smith bot. A specialist reporting machine
work is the second kind, because "a machine reporting machine work does not
wear a person's name". So slacker shells out to the one bot poster binary,
through the same SMITH_SLACK_POST env seam task-event:93 uses so a test can
never post for real, and contains no MCP call, no bot token and no third poster.
⚠️ IT IS TRIGGERED BY THE UPDATER'S SIGNAL AND READS THE BOARD ANYWAY.
the-specialists.md says to trigger it off the updater, "because the updater
holds the only reliable signal: a row flipping to observably-done". That is
true about the SIGNAL and not about the CLOCK. updater writes a prs ref; the
row's status DERIVES from that ref when propflow-docs/bin/refresh-tracker runs
from launchd, up to 300s later. So at the instant updater exits 0 the row
still reads open, and a slacker wired to that exit would announce nothing,
ever. It sweeps the published dock instead and compares it to what it last saw —
no hook in updater was needed and none was added, and it does not
re-derive "is this really done", which is where the roster says things drift.
⚠️ ZERO AUTOMATED CALLERS as of 2026-09-12 — the same state as cleaner and
optimizer, and stated rather than implied. It runs when somebody runs it. The
natural caller is a refresher tick, not updater's exit, for the reason above.
What the receipt is for, precisely. Re-running the same sweep is already a
no-op without it, because the state file has advanced — so the receipt is the
guard for the cases where the state has NOT: a run killed between the post and
the end-of-run state write, two copies started a second apart, a state file
restored from a backup. tests/slacker-the-same-closure-is-announced-once.py
makes that the sharpest case by rolling the state file back after a successful
post and asserting the second run is stopped by the RECEIPT. Six sabotage
controls, each run and each proven to redden exactly the case it names; a
seventh is recorded as reddening NOTHING, with the reason — the baseline
early-return and the was is None guard hold the same property, and a reader
who tries it deserves to know which one does the work.
Appendix — everything linked from this entry
red-main
| What it does | Held ONE atomic claim naming the single fixing lane when propflowai's push:main CI went red — minted by the Supervisor or adopted from a human's /driver lane — so every other fixer (Smith's alert lane, a second session, a person) could read who owned it and stand down |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agentflow |
| Deployed at | nowhere since 2026-10-18 — it ran as supervisor/src/redmain.ts inside com.propflow.agentflow-supervisor (recoverable at d77b030) |
| Status | superseded → main-status (ADR-0074, 2026-10-27): the red-main record is now a declaration a person or session writes with `main-status red` / `green`. Between the Supervisor's deletion (ADR-0067) and that, nothing wrote the claim and Agent Smith's channel-alert lane found no live owner |
| Live on the Systems tab | propflowai.co/agents → Systems → red-main |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
⚠️ SUPERSEDED 2026-10-18
(ADR-0067). This was a phase of
the Supervisor, and the Supervisor is deleted. Nothing takes the claim or mints
a fixer now. The relay's GET /api/red-main still reads the claim file, which
is normally absent, and says nobody holds it. Agent Smith's channel-alert lane
finds no live owner and opens its own fix. Everything below describes how it
worked until then, and is kept as history.
What it is. One phase of the supervisor sweep (superviseRedMain, every
60s). It reads the Alert on red main job of the latest push:main run; on red
it opens an incident (~/.claude/jobs/red-main/incident.json, one per
stretch of main being broken, closed only by main going green) and takes the
claim (claim.json, a flock-guarded compare-and-swap in
bin/redmain-claim.py) naming the single owning task, then mints that task
via start-operator — or, since 2026-09-10, adopts a live red-main-*
task a human minted via /driver or /operator after the incident opened. Every tick the
claim is held it logs red-main owned by <task> since <ISO> to that task's
supervisor.log and to the daemon log. Released when main is green or the
owning task is done/abandoned; orphaned claims are reaped by TTL.
parked does NOT release the claim (CLAIM_RELEASING_TASK_STATES, since
2026-09-12). A fixer that raises a block and parks is a fixer that is still
mid-fix — with a branch, and very possibly an open PR — so taking its red away
would mint a rival beside a fix in flight, the exact ADR-0015 race. The claim is
HELD and heartbeaten, a red-main-owner-parked alarm goes to a person, and a
parked red-main-* lane is still ADOPTABLE for the same reason — unless it
carries no operatorSid, which is how intake.ts marks a task whose session was
torn down: adopting that corpse would hold the claim forever behind nobody. ⚠️ Anything
still saying otherwise is stale, including ~/.local/bin/canon/fixer-SKILL.md
§4.4, which tells every appointed fixer never to park because parking "hands
your red to a rival mint".
The fixer MERGES its own fix (since 2026-09-12; Gera, "it could merge a PR
and everything… just to make sure that everything is clean"). The goal file used
to say "Open a PR. Never merge — that is a human decision" while the criterion
made the fixer the owner until push:main was green — a job plus a prohibition
on the only step that reaches it — and --allow-push had been passed at the mint
the whole time, so only the sentence was in the way. Both strings now grant the
merge and scope it: this red's PR, not the feature work that caused the red,
and no refactors on the way past. A hold-for-review label is still a stop sign.
A door refusal is not a consumed owner. start-operator exit 75 is
session-cap refusing the start because the box is at its ceiling — nothing was
launched. It used to end the claim as released, which the incident's
MAX_MINT_ATTEMPTS counter counted, so two refusals ninety seconds apart spent
the whole budget and a human was told two owners had failed at a red nobody had
been assigned to. Refusals now end the claim as refused, are excluded from the
cap, back off (MINT_REFUSAL_BACKOFF_MS, 5m) instead of knocking every sweep,
and raise their own alarm — red-main-mint-refused, the machine is full, free a
slot — while the phase keeps trying for as long as main is red.
It announces, as a SIGNAL. One write to one shared surface when a fixer is
confirmed to own the red (announced on the Driver BIND, never on the bare
claim — ~30% of claims never produced a session) and one when main goes green.
Rendered from claim.json + incident.json, which are already the record; it
adds no new source of truth and messages nobody (ADR-0003: signals, never
calls). The surface is #agent-smith via agent_smith.slack_post on stdin —
Smith's existing poster, shelled out to like start-operator, no Slack code in
this repo. Off if that interpreter is absent; the claim is still the record.
What the signals read, and why not claim.json. status.json is
rewritten after every sweep, red or green, claimed or not — {at, red, sha, incident, ownedBy, ownedSince, driverSid, heartbeatAt} — so its age proves
the phase ran and its body says who owns the current red. claim.json cannot
be the liveness signal: it is absent for the whole of a green day, and absent
must never read as dead. Five missed sweeps (300s) and the row goes Stale; the
signal and the detail block name the same file and the same 300s on purpose
and must move together (the ccswitch rule). The launchd signal is the
daemon itself: the phase cannot tick if its host is not loaded. ⚠️ The
reason given here used to be "the daemon log is ISO-stamped, which the
log-match-age parser does not read", and agentflow-relay#349 made that
false — the parser reads …T…Z now. The signal stays a file-age on the
stronger reason that was always the real one: status.json is rewritten after
EVERY sweep, red or green, so its age proves the phase ran, while the daemon
log's terminal line appears only on some paths.
Who reads the claim. Agent Smith's channel-alert lane reads claim.json
incident.jsonbefore it opens asmith-fix/*-push-main-red-*branch and stands down in the alert thread when a live owner exists (agent-smithred_main_owner.py).supervise --red-mainprintsstatusLine(). The 2026-09-10 incident this exists for: both supervisor mints were refused at the door (session cap), the refusals spent the mint budget, Gera minted the fixing lane himself, and — with no claim on disk — Smith and a human each opened a rival PR within two minutes of the owner's.
Appendix — everything linked from this entry
main-status (one red/green declaration of propflowai main)
| What it does | One command declares propflowai's main red (`main-status red`, naming the owner and the failing sha) or green (`main-status green`, refused unless the newest completed push CI run on main succeeded), and prints the current state with its CI evidence (`main-status`). The record is ~/.claude/jobs/main-status/state.json plus an append-only history.jsonl. Agent Smith's alert lane reads it first and stands down on a declared red, which ends on green evidence and never on a clock |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent-harness src/agent_harness/main_status.py (console script `main-status`), read by red_main_owner.read_declared_owner inside Smith's worker |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → main-status (one red/green declaration of propflowai main) |
Liveness signals
| Signal | How it is read |
|---|---|
| declared this week | file-age · ~/.claude/jobs/main-status/history.jsonl · stale after 10080 min |
Why it exists (2026-10-27). Gera refused a 120-minute timeout on a red-main claim with no end time (agent-harness #1465): with a timer, everyone waits the full 120 minutes. Instead, whoever fixes main says so, and green is checked against main's real push CI. ADR-0074.
Reading the signal. history.jsonl gets one line per declaration and
nothing else, and nothing writes it on a schedule. The 7-day window is a
judgement, not a measured red rate. Seven days without a line most likely
means the fleet stopped declaring. Look at how the last reds were handled
before reading it as a broken tool. An
absent state.json reads as nothing declared, never as green. proposed
until part 1 merges and the first real declaration lands; then this flips to
active.
Part 2 (ADR-0074, What is left): the bridge post in the id:/driver:
BRIDGE CLAIM v1 shape Trinity already reads, msg --all to the mini's live
sessions, folding Trinity's claims and RESOLVED into the same record, and
retiring the dead red-main claim reader.
Appendix — everything linked from this entry
agent-broadcast (one tell-everybody command)
| What it does | One command (`agent-broadcast --body-file F`) tells everybody one thing: every live mini session through `msg --all --json`, Smith through an append-only ledger (~/.claude/jobs/agent-broadcast/ledger.jsonl) that prompts.build_system_prompt reads into every brain turn while a notice is active (6h), and the humans in the engineering bridge through one `AGENT BROADCAST v1` post (never paging a human, sealed, burst-gated, refused if it would wake the other seat). The outcome is read only from msg's machine-readable receipt line, never its prose: reached / refused before sending (safe to re-send) / unknown (never re-send), capped at 90s under a session's tool call. The exit code says whether re-running the whole command is safe; a refused door prints a re-send for itself alone (`--only sessions|bridge`). main-status red/green will call it |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent-harness src/agent_harness/agent_broadcast.py (console script `agent-broadcast`, on PATH via local-bin's launcher) agent_broadcast_ledger.py, whose reader runs inside Smith's worker on every prompt build, and agent_broadcast_bridge.py (the bridge door) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → agent-broadcast (one tell-everybody command) |
Liveness signals
| Signal | How it is read |
|---|---|
| Smith's ledger reader ran today | file-age · ~/.claude/jobs/agent-broadcast/reader.status · stale after 1440 min |
Why it exists (2026-10-27). Gera, after ruling out a 120-minute red-main timer: whoever fixes main should "broadcast something … saying main is green, continue work, and that somehow messages all the sessions". He then asked for it general: one command that tells everybody. ADR-0074.
Reading the signal. The reader is agent_broadcast_ledger.active, called
by prompts.build_system_prompt on every Smith brain turn. Every call rewrites
reader.status ({read_at, active}), whether or not anything is active. Smith
takes turns all day, so a status file older than a day means the reader is not
running inside Smith (a deploy without it, or prompt builds failing). It does
NOT mean nobody broadcast. To see what Smith is being told, read
ledger.jsonl's lines whose expires_at is still in the future.
Doors. Sessions (msg --all --json), Smith (the ledger) and the
engineering bridge are live. The bridge door tells the HUMANS there: Trinity's
harness drops a bridge post that does not @-mention it, with no record, and
the door refuses a notice that would wake Trinity, because whether a broadcast
reaches Trinity is Fede's call (ADR-0074's ask). The bridge door is dark in any
shell without SLACK_BRIDGE_CHANNEL; every claude session on the mini
carries it (measured 2026-10-28, 43 of 43).
Appendix — everything linked from this entry
blocker-trap (a helper's self-fixable blocker gets a fix owner the same turn)
| What it does | Greps every helper report as it arrives (the <result> of a <task-notification> on UserPromptSubmit, a foreground Agent result on PostToolUse) for a blocker report: a strong phrase (login rejected, `STUCK <name>:`, couldn't … from here, stop trying) or a symptom beside a giving-up phrase. On a hit it injects a reminder that stage, local, dev-server, test-account, CI-flake and credential-we-hold problems are ours; at Stop it blocks ONCE if the turn neither started a helper nor named a true-hold category (prod write / money / customer send / irreversible / product decision / harness denial). No model call, no daemon, fails open. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | ~/.local/bin/canon/blocker-trap.py, linked to ~/.claude/hooks/blocker-trap.py and registered under UserPromptSubmit, PostToolUse (Agent) and Stop by loop-doctor |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → blocker-trap (a helper's self-fixable blocker gets a fix owner the same turn) |
Liveness signals
| Signal | How it is read |
|---|---|
| firing | log-match-age · ~/.claude/jobs/blocker-trap/fires.log · matches FIRE · stale after 4320 min |
Why it exists (2026-10-25). A scan of 1,727 transcripts since 10-18 found that about half of all stop-short moments were self-fixable. The stage smoke login stayed broken 7 days across 8 sessions, and once a helper tried, the fix took 4 minutes and touched only stage. The text that let lanes hold without trying had been pasted into briefs 103 times across 29 sessions. ADR-0073.
Reading the signal. fires.log gets one line per fire, block and pass, and nothing when nothing fires. Replayed over the week to 10-25, the hook fired on 124 of 10,813 helper reports (about 18 a day in 21 sessions), so 3 days without a FIRE most likely means it is not registered or not loaded. Check that before reading it as a quiet week: loop-doctor --check should show blocker report, blocker result and blocker stop as registered. A line reading BLOCK means a coordinator was stopped once. PASS helper-started and PASS true-hold-named are turns that answered the report on their own.
Appendix — everything linked from this entry
nudge (idle watchers)
| What it does | Reflex — wakes a parked session after N seconds of stillness while it has work or an open block, never faster than once per 600s (ADR-0068); self-compacts at 80% with a derived keep-list; a say-something-new gate with a 24h TTL; `--retire-checkin` stops a session's retired Driver check-in watcher(s) for watcher-doctor |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/nudge |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → nudge (idle watchers) |
Liveness signals
| Signal | How it is read |
|---|---|
| nudge.log written today | file-age · ~/.claude/jobs/nudge/nudge.log · stale after 1440 min |
| a wake delivered today | log-match-age · ~/.claude/jobs/nudge/nudge.log · matches NUDGED · stale after 1440 min |
Rung 2 of the ladder. One --_watch process per armed session, registered by
pid file under ~/.claude/jobs/nudge/. It is the only thing that can resume a
stopped session (a Stop hook cannot), so its death is silent by construction —
which is why watcher-doctor below exists. Since local-bin#37 a watcher whose
block state has not moved speaks once, escalates at 1h and 6h with different
text, and retires at 24h; a decision link rides every blocked wake (#39). From
local-bin#260/#262 (2026-09-24) until ADR-0068 (2026-10-18) every live Driver got
the same Driver check-in, armed by watcher-doctor and by start-driver at
spawn; that check-in is retired — see the ADR-0068 paragraph below, which also
says why both signals above are now a day long. A Driver's watcher, where a
session arms one for itself, still goes silent past 24h instead of retiring.
A watcher exits by itself when its session has no live host (2026-10-03,
ADR-0040 §4, local-bin #377). Measured that morning: 12 --_watch
processes reparented to launchd, up to 3.5 days old; the roster exit ("target
gone") never fires for a session the vendor roster still lists as resumable,
which is every session whose host simply exited. Each wake the watcher asks
fleet-medic host-alive <sid> (the lease + process-table answer, as a word:
ALIVE / DEAD / UNKNOWN; bounded at 30 s); DEAD on two reads at least 900 s
apart (three medic intervals, and the 15 min liveness still calls a
roster-less session with a fresh transcript ALIVE) removes its pid file, logs
watch exit — sid <8> has no live host and exits 0; ALIVE in between resets;
UNKNOWN counts for nothing (the 2026-08-08 rule) and is logged as
HOST-ALIVE UNKNOWN … fleet-medic said: … on the first and each doubling.
Nothing kills a watcher from outside. Measured at deploy (local-bin #377,
16:45Z): all 15 live watchers targeted a sid whose lease holder was alive, so
none exited — the guard is for the next one. Watchers already running keep the
script they started with; the rule applies to every watcher armed after
deploy.
The Driver check-in is retired (ADR-0068, 2026-10-18). Until then
every live Driver got THE Driver check-in, a --_watch armed by watcher-doctor
and start-driver (ADR-0030 row 4; ADR-0067
had kept it as the Driver waker when the Supervisor's idlewake.ts went). Measured
2026-10-11..18: 571 check-ins, 7 % followed by any work, 33 % into a PAUSED
session, 57 % fired on this tool's 60s answer-poll floor instead of the 600s idle
rule. Now fleet-medic is the one nudger (it pokes only a session the session pulse
reads needs-kickoff), nothing arms a check-in, and watcher-doctor --repair
retires each live one through nudge --retire-checkin --sid <sid> — only that
session's check-in watcher(s), never its own one-shot or --every wakes. Both 60s
floors are 600s: NUDGE_MIN_INTERVAL (the shortest --idle/--every it will
arm; one 60s watcher put 881 prompts into one session in three days) and the
open-block poll. Both signals above are therefore a day, not 30 minutes and 6
hours: the per-Driver check-ins were most of what wrote nudge.log (142
NUDGED on 2026-10-18 alone), and with them retired a healthy fleet can go hours
with no watcher ticking and no wake delivered — fewer wakes is the point. A day
of silence still means the tool, or every session's use of it, is gone.
Appendix — everything linked from this entry
watcher-doctor
| What it does | Reflex — every 180s, RETIRES every live Driver nudge check-in (ADR-0068: fleet-medic's pulse-gated poke is the one nudger) and arms nothing; reports each live Driver as ARMED (its own wake), UNWATCHED (no watcher, the designed state), STOOD-DOWN or CHECK-IN; reaps orphaned watcher state files |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/watcher-doctor |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → watcher-doctor |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.watcher-doctor |
| sweep completed | log-match-age · ~/.claude/logs/watcher-doctor.log · matches expected lane(s) · stale after 15 min |
| no Driver check-in left | log-match-age · ~/.claude/logs/watcher-doctor.log · matches · 0 check-in(s) left · · stale after 30 min |
Asks the PROCESS TABLE, never the corpse: a watcher only ever records the
death it chose. Since local-bin#38 a dark lane with no stored arm record is
re-armed from start-operator's own wake text derived from task.json — never
invented — and the fleet-wide killswitch (~/.claude/nudge-rearm.OFF) prints
its age and contents in every refusal, because an empty marker held 27 lanes
dark for eleven days with nobody able to tell whether it was still deliberate.
Since local-bin#260 (2026-09-24, ADR-0030 row 4) EXPECTED means every live
Driver. A parked task's Driver counts while its session is alive, and a
running task-driver workflow no longer excuses the check-in, so the
CLOUD / CLOUD+BASH verdicts are gone. Before that, the Supervisor's parked
and a running workflow each hid a live Driver, and 4 of 4 were unarmed.
--repair armed THE Driver check-in (watcher-doctor --checkin-text <task>)
and never replayed an arm record.
Since ADR-0068 (2026-10-18) it arms nothing. The check-in is retired
(fleet-medic pokes a session only while the session pulse reads it
needs-kickoff), and --repair stops every live one it finds — a PAUSED
Driver's included — through nudge --retire-checkin --sid <sid>. A watcher
keeps the script it was armed with, so the pre-ADR-0068 check-ins can only be
ended from outside, and this sweep is that outside. DARK is gone: a live Driver
with no watcher is UNWATCHED, the designed state, never repaired and never
alarmed. no Driver check-in left matches a sweep footer reading · 0 check-in(s) left ·, so it goes stale exactly while a check-in survives its
retirement (or --repair stops running); a retirement that fails is also exit 2.
Appendix — everything linked from this entry
fleet-medic
| What it does | Reflex — every 300s, heals stuck tmux-hosted Claude sessions on the session pulse (ADR-0068): relaunches a dead host only if its last pulse state was working or needs-kickoff (never paused/waiting/done/unknown or PAUSED; after a reboot working first, capped, under a load gate), kills and relaunches a wedged host, and is the one nudger — pokes only a needs-kickoff session, once per 10 min, two pokes without work then nothing until a human types; capped, logged with evidence, alarmed when it cannot heal |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fleet-medic (launchd co.propflow.fleet-medic, 300s); SessionEnd marker writer `fleet-medic session-end` (dotclaude settings.json); state + log ~/.claude/jobs/medic/; kill switch ~/.claude/jobs/medic/STOP |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → fleet-medic |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.fleet-medic |
| tick ran | file-age · ~/.claude/jobs/medic/heartbeat · stale after 15 min |
| nothing left unhealed | log-match-age · ~/.claude/jobs/medic/log · matches medic OK · stale after 20 min |
| reads a fresh pulse | log-match-age · ~/.claude/jobs/medic/log · matches pulse=fresh · stale after 30 min |
This exists because Gera was the detector. On 2026-09-24 four kinds of stuck
session each needed him to notice by eye: a mass kill left leased sessions with
no host and nothing restarted a tmux-hosted one; a Claude Code bug wedged hosts
at 0% CPU with unreaped hook children and an Esc that did nothing; a session
sat on a spent seat that ccswitch never moved; and sessions idled with nobody to
nudge them. His words: "we should fix both the root and the symptom, so if it
does happen we fix, but also, we fix the issue." The one-night
~/.claude/jobs/overnight-watch/watch.py (four sessions, idle nudges only, no
catalog entry, no schedule) is superseded by this and retired.
It decides on the session pulse, never on a name (ADR-0068, 2026-10-18).
Every tick it reads ~/.claude/jobs/pulse/snapshot.json (the session-pulse entry
below — a file, never a call) and remembers, in its own state.json, the state it
read for each session while its host was alive. A dead host is relaunched only if
that remembered state was working or needs-kickoff (config revive_states), it
is not PAUSED, and the current snapshot is fresh and does not show it running. A
lease last renewed before kern.boottime is a reboot orphan: working first, at
most 2 a tick, only while the load average is under the CPU count (on 2026-10-18 the
old rule relaunched 64 sessions after one reboot, load 45). An idle session is poked
only while the pulse reads it needs-kickoff, once per 600s, and two pokes in a row
with no work (no tool use beyond reads) stop it until a HUMAN prompt appears in its
transcript. A snapshot that is missing, stale, of another version or blind revives
and pokes NOTHING, and the tick says so (pulse=UNKNOWN) — which is what turns
reads a fresh pulse stale. The watch list and the seat move are gone (the
latter to ccswitch's walled-heal: 312 moves, 972 refusals, 4 % followed by work).
fleet-medic explain prints every verdict read-only.
It works only on what the lease proves. Scope is every conversation lease
(~/.claude/jobs/lease/<sid>.json) on this host with host_kind: tmux. Holder
liveness is DERIVED (pid + start time), never read from the file. A dead holder is
resurrected only if the medic remembers its pulse state from while it was alive —
historical leases are history, not outages — and only when the session did not
end deliberately. Relaunch goes through
ccswitch exec --session <name> (verified credential, twin refusal) and the
lease refuses a second copy, so a relaunch cannot make a duplicate.
⚠️ DELIBERATE CLOSE vs DEATH — the discriminator, measured. fleet-medic session-end (a SessionEnd hook) records the reason and whether the tmux pane
still existed. Measured with real Claude Code 2.1.281 hosts in a private tmux
server: /exit → prompt_input_exit; killing the window → other with the
pane already gone from list-panes -a; kill -TERM → other with the pane
still listed; kill -KILL → no hook at all. Only the last two are
resurrected. Known limit: tmux kill-server reads as every window closed on
purpose. And because Claude Code snapshots hooks at session start, a host that was
already running when the hook went live cannot write a marker at all: for those
(older than ~/.claude/jobs/medic/hook-since) "no marker" is unknowable, and
the medic alarms instead of relaunching.
⚠️ A WEDGE IS PROVEN, NOT GUESSED, AND AN IDLE HOST CANNOT BE PROVEN WEDGED.
The kill needs all of: a turn in progress on a screen frozen ≥10 min (spinner
timer included), zero CPU time between ticks, a <defunct> child born after the
transcript's last write (healthy hosts carry startup zombies for days —
measured on the mini — so an old zombie proves nothing), a silent transcript,
and one Esc that does not redraw the screen. Esc at an idle prompt changes
nothing on a HEALTHY host (measured), so wedge signs without a turn in progress
raise an alarm instead of a kill.
The seat move is retired (ADR-0068). The medic used to run ccswitch move
for a session stuck on a spent seat; ccswitch's walled-heal now owns every account
move, so one tool holds the job (two holders is the 2026-09-11 incident).
Signals. tick ran is a heartbeat touched by every real tick (a STOPPED
medic still ticks). nothing left unhealed keys on the literal medic OK,
written only when no alarm is open — a cap hit, a refused move, or wedge signs
it may not act on write medic ALARM instead, and the kill switch writes
medic STOPPED — so this row, and the jobs probe on the status line, go stale
exactly while something needs a human. Alarms also post once per 6h per cause
to the mini's own alert destination (mini-alerts.json) through the one bot
poster, plus a desktop notification.
Teardown. Oversight sensor: it is not the watched sessions' to switch off.
touch ~/.claude/jobs/medic/STOP pauses it; retiring it is a catalog change.
Appendix — everything linked from this entry
fleet-bake (week-long bake)
| What it does | Pipeline — every hour, measures the fleet's credential + wake machinery (panes, lanes, every catalog entry's state, ccswitch accounts, wakes, rotations, the last e2e PASS, the last controls run) and renders the week to a bot-owned Slack canvas bookmarked in #alerts |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fleet-bake |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → fleet-bake (week-long bake) |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.fleet-bake |
| tick appended | file-age · ~/.claude/logs/fleet-bake.jsonl · stale after 120 min |
The instrument that watches the week AFTER a fix is declared. Every prior
"fixed" on the credential machinery was declared from a moment; Gera asked on
2026-08-31 for something behind the scenes that, a week later, says what
worked and what regressed. fleet-bake tick (launchd
co.propflow.fleet-bake, StartInterval 3600) collects one record — all
derived, nothing remembered — appends it as one line to
~/.claude/logs/fleet-bake.jsonl, and re-renders the canvas from the whole
log. Read-only against the fleet: every unreachable source prints as a
regression, never as a blank that reads green.
The tick appended signal is 7200s — two ticks — so one slow hour on a busy
box is not a red row, and a job that has stopped appending is visible within
the hour after that. The systems section of every tick is this catalog's
own /api/systems answer, so the bake and the Systems tab can never disagree
about a row without one of them being provably wrong.
Appendix — everything linked from this entry
fleet-e2e (PITFALLS §0 as a command)
| What it does | Pipeline — the acceptance test for any credential change: spawns a FRESH session through `ccswitch exec`, requires a real turn (assistant message + tool call, nonce round-tripped) AND a session registration, then tears it down; writes the PASS marker the bake reads |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fleet-e2e |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → fleet-e2e (PITFALLS §0 as a command) |
Liveness signals
| Signal | How it is read |
|---|---|
| PASS within 7 days | file-age · ~/.claude/logs/fleet-e2e.last · stale after 10080 min |
PITFALLS §0: "a cap fix counts as fixed only when a previously-stranded live
session is observed completing a real turn on the healthy account. Not when
the symlink moved. Not when ccswitch list looked clean." Every one of the
~30 "fixed" declarations before 2026-08-31 would have failed that test, and
none of them ran it, because it was a paragraph and not a command. A turn is
a new assistant message PLUS a tool call — a 0s completion is a 429
rejection, a resume banner is not a turn, a live pid is not a turn — so the
command asks the session to echo a nonce through a shell tool and waits for
the nonce to come back through the tool result, the one shape a rejected
turn cannot fake. On PASS it writes ~/.claude/logs/fleet-e2e.last, a JSON
{at, email, cwd}: when, which account the gate bound, and where.
Run by hand, never on a timer, and the signal is deliberately long. A
run costs a real session start (~130K tokens of context on whichever account
the spawn gate binds), so it runs after a credential change — a rotation, a
pool edit, a ccswitch deploy — and not on a schedule. staleAfterS is
therefore 604800 (7 days): the same rule the bake applies ("no fleet-e2e
PASS in 7 days" is one of its regression rules), and a week without a PASS
means a week in which no credential change was accepted the way §0 demands,
which is worth a stale row. A shorter window would make the catalog nag for
a session start nobody needs. pipeline is the nearest kind the vocabulary
has for a check that is run, not a thing that runs — it is not a reflex
(nothing triggers it), not orchestration, not an actor.
Appendix — everything linked from this entry
fleet-controls (weekly control run)
| What it does | Pipeline — every Sunday 04:00, runs every local-bin control script and every ccswitch suite under a timeout and appends one record per run — passed / failed / could-not-grade / not-run / flaky-passed, with the failing names — for the bake to read |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fleet-controls |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → fleet-controls (weekly control run) |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.fleet-controls |
| a run recorded within 8 days | file-age · ~/.claude/logs/fleet-controls.jsonl · stale after 11520 min |
Until 2026-09-01 the controls (~/.local/bin/tests/*.sh, 56 scripts, and
~/code/ccswitch/tests/*.py, 36 suites) ran only when a person ran them
from memory: no CI, no cron, no hook. A control that runs only when someone
remembers stops running the week after it was written, and a spec nobody
runs is a document. launchd co.propflow.fleet-controls runs the whole set
weekly (StartCalendarInterval Sunday 04:00, when the fleet is quietest —
control-35 arms real watchers and the set takes 10–15 minutes; RunAtLoad
false so a bootstrap never starts a 15-minute run on a busy box) and appends
one line to ~/.claude/logs/fleet-controls.jsonl. The exit code reports the
instrument, not the finding — 0 means the run completed and was recorded —
so launchd's LastExitStatus says whether the JOB ran and the verdict lives
in the jsonl, which is why the launchd signal here cannot turn a failed
suite into a red job, on purpose. The bake reads the last record into its
"Right now" table and its two regression rules, "controls failed" and "no
controls run in 8 days"; staleAfterS 691200 (8 days) is that same rule —
one weekly slot plus a day of slack for a run that starts late.
First run, 2026-09-01: 73 passed / 21 failed / 6 not run across 100
suites (763s). The 21 are pre-existing reds on the deployed tools — they
were red the day before, and the day before that, invisibly — now visible
weekly instead of never. Their names are in the record; fixing them is
local-bin work, and this row does not go green by hiding them. Two ccswitch
suites (keychain-and-panes, wedge-heal) fail in a batch and pass alone;
a failure there is rerun once, alone, and counted as flaky-passed if green —
never as a silent pass and never as a fail that was really the batch.
Appendix — everything linked from this entry
relay-deploy (the live branch, advanced safely)
| What it does | Pipeline — every 15 minutes, fast-forwards the relay's `live` branch to origin/main when every incoming file is one the running process reads at request time, and HOLDS the moment main carries code that would need a build |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/relay-deploy |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → relay-deploy (the live branch, advanced safely) |
Liveness signals
| Signal | How it is read |
|---|---|
| sweeper scheduled | launchd · co.propflow.relay-deploy |
| sweep succeeding | log-match-age · ~/.claude/jobs/relay-deploy/relay-deploy.log · matches SWEEP OK · stale after 45 min |
| reading fresh | file-age · ~/.claude/jobs/relay-deploy/snapshot.json · stale after 45 min |
Detail document: ~/.claude/jobs/relay-deploy/snapshot.json (the Systems tab opens this row into it).
Merging a PR to main does not deploy this relay. It runs from the live
branch of ~/code/PropFlow/agentflow, and until 2026-09-02 nothing advanced
it — the Systems tab was found serving a catalog two commits behind, with
#96 merged
and simply never live. Nobody knew, because a stale catalog looks exactly like
a current one.
⚠️ This is deliberately not git merge --ff-only on a timer. relay/dist
(and supervisor/dist, until ADR-0067 deleted the Supervisor) is
gitignored, and launchd execs the absolute dist path, so the two halves of this repo deploy on completely different rules:
| how it reaches the running process | what a fast-forward does | |
|---|---|---|
SYSTEMS.md, docs/ |
parsed at request time | deploys it instantly |
.ts source |
compiled into gitignored dist/ |
nothing — inert until a rebuild |
A blind timer therefore does something worse than nothing: it walks the source
ahead of the running binary, so the next person to run npm run build in that
checkout ships every accumulated commit at a moment they did not choose and
never reviewed. That is the inverted hazard — git operations are inert, BUILD
operations are destructive — and putting it on a schedule loads the gun
automatically.
So the job ships only what the running process reads at request time
(default-deny allowlist: SYSTEMS.md, README.md, docs/) and holds the
moment main carries anything else, naming what it held in the snapshot. One
code file holds the whole batch, deliberately — a catalog edit riding behind a
source change is not a safe deploy just because the catalog half is. A human
sequences those, because they need a build and a restart.
Fail-safe by construction: it refuses a dirty tree, refuses a checkout that is
not on live (that is somebody's in-flight work, not an invitation), refuses a
diverged branch, and only ever runs git merge --ff-only. It never builds,
never restarts, never force-anythings.
A held deploy is information, not a fault, so it does not colour the row
red — the log line opens SWEEP OK whenever the check itself ran, SWEEP FAIL
only when it could not run at all, and what is waiting on a human shows in the
held block of the detail panel.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
fleet-deploy (seat versions, and the seats nobody can read)
| What it does | Reads the boot record every agent SEAT emits at worker start, compares each one to main, and reports which seat is running which commit — naming any seat this host cannot read as UNKNOWN rather than letting its absence read as up to date |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | smith-fleet-deploy (agent_smith/fleet_deploy.py) |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → fleet-deploy (seat versions, and the seats nobody can read) |
Liveness signals
| Signal | How it is read |
|---|---|
| reader completing a pass | log-match-age · ~/.claude/jobs/fleet-deploy/fleet-deploy.log · matches FLEET-DEPLOY · stale after 65 min |
| roll-up fresh | file-age · ~/.claude/jobs/fleet-deploy/snapshot.json · stale after 65 min |
Detail document: ~/.claude/jobs/fleet-deploy/snapshot.json (the Systems tab opens this row into it).
relay-deploy above says a merge to main does not deploy the relay. This
row is the same sentence about the AGENT SEATS, and it cost more to learn.
agent-smith PR #376 fixed the false-amnesia banner ("I don't have this thread's earlier turns for this reply"), verified at zero occurrences from Smith's own
checkout. On 2026-09-07 between 02:39 and 02:44Z the clara seat — presenting
as Trinity — emitted that same banner three times: same codebase, different
AGENT_SLUG (ADR-0001), another machine, another checkout, and the fix had
never reached it.
Nothing on any surface could say so, and neither instrument was broken. Smith's
outbound ledger read 0 because it records posts made by the agent_smith
package on this box (the 63% blind spot agent-smith#399 documented) — a seat
on another host is not under-counted there, it is absent, and the absence
rendered as a zero. And the per-seat fact existed but had no cross-seat reader:
deploy_state.record_boot has every worker declare its commit at boot into its
own state dir, while the only consumer asked this_boot() — this process, this
host.
The rule this row enforces, and the one thing it will never say. A seat
whose boot record this host cannot read is UNKNOWN. Never live, and the
roll-up has no path to aligned while one exists — the coverage clause rides
the clean line as loudly as the loud one. Run on the mini, that reads:
FLEET-DEPLOY BEHIND main=d01d8d87 smith=7a52b553(-1) · clara=unknown
— seats read: smith of 2; NOT covered: clara
smith (Agent Smith): BEHIND — 7a52b5538 (1 commit behind main), booted 2m ago
clara (Trinity): UNKNOWN — no boot record at ~/.claude/clara-state/worker-boot.json
⚠️ Trinity will keep reading UNKNOWN from this host, and that is the honest
answer rather than a defect. Its record is written on its own machine and no
transport carries it here. The reader deliberately does not reach out to ask —
ADR-0003, rungs integrate by signals never by calls — so the supported closes
are to run smith-fleet-deploy on that host, or to point
CLARA_SMITH_STATE_DIR at a synced copy of its worker-boot.json. Updating a
seat on another machine is a human action; nothing in this system deploys
anything. ADR-0011
records why a general cross-host transport is a separate decision.
Why the signals prove the READER ran and not that the fleet is aligned. A
fleet being briefly behind is the normal state after every merge — seats pick up
new code when their workers restart, not when a PR lands. Keying this row on
ALIGNED would paint it red for every ordinary merge and train everyone to
ignore it, which is ADR-0009's "green means the thermometer works" failing in
the other direction. So FLEET-DEPLOY opens every completed pass, aligned and
behind and unknown alike, and the verdict lives in the detail panel — the same
split relay-deploy makes for a held deploy. A pattern that only appeared on
the unhappy path would make this row go stale exactly when everything was fine.
⚠️ The pattern is the literal FLEET-DEPLOY because the evaluator matches
SUBSTRINGS, not regexes (the HOST (OK|PRESSURE) lesson two rows up). The
stamp is a local YYYY-MM-DD HH:MM:SS; since agentflow-relay#349 an ISO
…T…Z or offset form is read too, so that is now the writer's choice rather
than the only readable shape — LOG_STAMP in relay/src/systems/catalog.ts is the authoritative statement of what the evaluator can date, and this entry does not restate
it. The stamp's being load-bearing at all is why agent-smith's
test_log_line_stamp_is_what_the_relay_can_parse asserts against the
evaluator's own regex rather than the writer's format string, and that test
still passes: the space form is still accepted.
proposed, not active, and the signals above are already the real ones.
agent-smith#415 is open and nothing schedules the reader yet, so both signals
render cannot read … → unknown, which is the truthful answer to "is this
alive": it is not. That is the session-pulse pattern below, deliberately —
the same declaration becomes the live canary with no edit, so there is no second
step anyone can forget and no window in which this entry claims liveness the
catalog cannot see. It flips to active when the PR lands and an hourly run
exists; staleAfterS: 3900 is the promise that cadence owes.
Appendix — everything linked from this entry
agentflow (Driver / relay)
| What it does | Orchestration — takes the DOCK it is assigned at setup, assigns its action items to sub-agents (parallel where the optimizer proves independence, serial where it does not), judges progress, and surfaces only the ~5% of forks a human must settle |
|---|---|
| Kind | orchestration |
| Repo | https://github.com/PropFlow-Technologies/agentflow |
| Deployed at | — |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → agentflow (Driver / relay) |
Liveness signals
| Signal | How it is read |
|---|---|
| relay serving | self |
| relay log written | file-age · ~/.claude/logs/agentflow-relay.log · stale after 10 min |
⚠️ THE SUPERVISOR WAS DELETED on 2026-10-18
(ADR-0067), the same day it was
switched off (ADR-0066). Gera's call:
nothing polling in the background, and the code stays in git (d77b030) if it
is ever wanted. This entry is now the Driver and the relay. Nothing revives or
mints a Driver; /agents no longer starts filed jobs (POST /api/intake and its
three siblings answer 410); a decision answered on a docs.propflowai.co page
reads answered only once the raiser runs blocked resolve (answers given on
/agents land directly); and the Supervisor's auto-compaction, mail wakes and
alarms are gone. The supervisor loaded signal went with it. ADR-0067 has the
full list and how to bring it back.
Rung 3 of the ladder is now the Driver alone: it judges its own work against a
criterion and escalates to the human (rung 4) via decision pages. Opt-in per
task — a session with no stated goal is deliberately outside its reach, which
is exactly why the reflex layer below it must not depend on it. Integration
between rungs is signals (files), never calls. What the Supervisor's final
code did (ADR-0035's one-waker/one-reviver split, #405's late-boot fix)
is recorded in ADR-0035,
ADR-0066 and the code at d77b030.
Appendix — everything linked from this entry
Session pulse (state classifier)
| What it does | One answer to "what is this session doing" — six states or null, with the wake path that decides between waiting and needs-kickoff |
|---|---|
| Kind | orchestration |
| Repo | https://github.com/PropFlow-Technologies/agentflow |
| Deployed at | relay/src/pulse/ |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Session pulse (state classifier) |
Liveness signals
| Signal | How it is read |
|---|---|
| pulse snapshot fresh | file-age · ~/.claude/jobs/pulse/snapshot.json · stale after 15 min |
active since the collector shipped — and the signal above is byte-for-byte
the one the entry carried while it was proposed. That was the whole design.
It sat as a file-age on a path nothing wrote, rendering unknown
(cannot read …), which was the truthful answer to "is this alive": it was
not. relay/src/pulse/collector.ts now writes that exact path every 120s, so
the same declaration became the real freshness canary with no edit — there was
no second step anyone could forget, and there is no window in which the entry
claimed liveness the catalog could not see.
⚠️ THE 120s CADENCE IS A PROMISE TO THE staleAfterS: 900 ABOVE, AND IT IS
TESTED, NOT ASSERTED. relay/tests/pulseCollector.test.ts parses this file,
finds this entry, and checks the collector's interval leaves a wide margin
inside the declared bound (seven consecutive missed cycles before the signal
goes stale — an outage, not a slow sweep). Its first assertion is that it
FOUND the entry, because a parser that matches nothing agrees with every
number. Change either side and that test is where you will hear about it.
⚠️ A self signal was the obvious alternative and is wrong here. self
evaluates to ok unconditionally (relay/src/systems/catalog.ts) — it is the
relay reporting that the relay is serving. On the relay's own entry that is
exactly right; on a module that merely SHIPS inside the relay it is an
unconditional green over code that may never be called, which is the
inert-mechanism failure ADR-0004's liveness clause exists to prevent. It would
also have gone green on the day this system was dead code, which is precisely
the day it needed to report otherwise.
It has a reader that ACTS on it (ADR-0068, 2026-10-18). fleet-medic
relaunches a dead host only if its last state here was working or
needs-kickoff, and pokes an idle session only while it reads needs-kickoff; a
missing, stale, blind or other-version document makes it act on nothing. It pins
PULSE_SNAPSHOT_VERSION 1, so a change to what a state MEANS must bump the version
— the medic then reads every snapshot as unknown and stops acting until it is
updated, which is the safe direction. Note for anyone changing the wake-path
rule: a live Driver is its own wake path here (driverBoundFor), so it reads
waiting, never needs-kickoff, and the medic therefore never pokes one.
What the collector does, in one paragraph. Every 120s it takes the roster
the relay already polls, captures each session's VISIBLE pane region (no -S
— the region discipline is the defence, not an optimisation), reads the DONE
markers, the block ledger, the task ledger and the phase arbiter ONCE for the
whole fleet, runs classify() per session, and stage-and-renames one document
to the path above. It writes that one file and nothing else; the only tmux
command on its path is capture-pane. A failed cycle deliberately writes
NOTHING, so the file ages and this signal says so — a degraded write would
refresh the mtime and report health that was not measured.
Rung 3 of the ladder, beside the Driver (the Supervisor sat here too, until
ADR-0067): it decides, it never acts.
Classification and actuation are separate layers on purpose — every
"nudged a session that shouldn't have been nudged" incident this replaces came
from one elif chain that did both. See ADR-0006 for the six states, the
wake-path rule, and the inventory of the five tools it absorbs.
Appendix — everything linked from this entry
Remote pool (AWS worker)
| What it does | Keeps our servers on Gera's dedicated subscription accounts, never shared with the fleet or the reviewers: probes the ccswitch pool claude-pool-remote and publishes its live bearers into all three server copies of ANTHROPIC_BACKGROUND_AUTH_TOKEN (the ECS worker's secret, the Lambdas' SSM parameter, Vercel production), each only when it changed and only if the fleet's pool does not also hold the account; writes the account status and the where-every-account-runs map the /agents ccswitch row shows. A step inside the reviewer-token cycle, not a job of its own |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/dotclaude |
| Deployed at | ~/.claude/scripts/remote-pool-cycle.py |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Remote pool (AWS worker) |
Liveness signals
| Signal | How it is read |
|---|---|
| cycle ran | log-match-age · ~/.claude/logs/remote-pool.log · matches remote pool (claude-pool-remote) · stale after 180 min |
| account status written | file-age · ~/.claude/jobs/remote-pool/status.json · stale after 180 min |
| pool can serve the worker | pool-live · ~/.claude/jobs/remote-pool/status.json · stale after 180 min |
Detail document: ~/.claude/jobs/remote-pool/status.json (the Systems tab opens this row into it).
Runs inside rotate-reviewer-tokens.sh's 2-hour cycle rather than as a job of
its own (Gera, 2026-10-18: no new polling jobs that spend tokens), at one
1-token probe per pooled account. The pool, the secret the worker reads, and
why there is exactly one account are in the ccswitch section above. The status
document has the same shape as the reviewers' (version, accounts[] with
email, state, util_5h, util_7d), plus publish (published,
in-sync, would-publish on a dry run, refused or skipped, with the
reason), so pool-live reads it
unchanged.
Appendix
- https://github.com/gera-propflow/dotclaude#readme — the README (no docs declared in SYSTEMS.md yet; add some)
Reviewer token pipeline
| What it does | Keeps CI code-review authenticated across all repos from the reviewers' OWN pool (AWS claude-pool-reviewer, never the fleet's bank) — probes it, publishes its primary accounts first and secondaries only behind them (worst-axis headroom orders within a tier), writes GitHub secrets every 2h; an unreadable pool publishes nothing and the secrets keep their last values |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/dotclaude |
| Deployed at | ~/.claude/scripts/rotate-reviewer-tokens.sh |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Reviewer token pipeline |
Liveness signals
| Signal | How it is read |
|---|---|
| cycle completed recently | log-match-age · ~/.claude/logs/reviewer-token-rotate.log · matches cycle done · stale after 180 min |
| fed from its own pool | log-match-age · ~/.claude/logs/reviewer-token-rotate.log · matches reviewer rank (claude-pool-reviewer) · stale after 180 min |
| outside-write check ran | log-match-age · ~/.claude/logs/reviewer-token-rotate.log · matches pool-writer check exit · stale after 180 min |
| launchd job loaded | launchd · com.propflow.reviewer-token-rotate |
| account status written | file-age · ~/.claude/jobs/reviewer-pool/status.json · stale after 180 min |
| pool can serve a review | pool-live · ~/.claude/jobs/reviewer-pool/status.json · stale after 180 min |
Detail document: ~/.claude/jobs/reviewer-pool/status.json (the Systems tab opens this row into it).
The cycle done signal is the freshness canary this pipeline earned on
2026-08-29: the job died silently every 2 hours for ~13 hours (launchd ran it
under macOS /bin/bash 3.2, which lacks declare -A) and its only voice was
itself — dead things don't speak. It is the external witness OF THE ROTATOR,
and that is the whole of what it witnesses — the sentence here used to stop
at "the external witness", which read as a claim about the pipeline. Dead
things don't speak; a capped pool speaks perfectly, on time, every two hours.
Which is why this row has a sixth signal, and it is the only one that can go
red while the rotator is healthy. On 2026-10-26 the code-review bot was
unusable across three repos for about three hours: every account in
claude-pool-reviewer answered HTTP 429, and six finished PRs plus three in
other repos sat with no verdict, each surfacing it per-PR as Review infra failure (tool_crash) and All 3 Claude account slots are OUT OF BUDGET (HTTP 429). This row was GREEN for all of it. Measured off
~/.claude/logs/reviewer-token-rotate.log, five consecutive cycles from
2026-10-25 20:19 to 2026-10-26 04:33 local: every one logged reviewer cap … 429 three times, wrote a fresh status document, and closed with cycle done —
satisfying the three log markers, the launchd probe and the file-age signal at
once. The per-account state was recorded and rendered the whole time; nothing
read it. pool can serve a review reads it.
⚠️ cycle done's own numbers are not this pool's, and the one ALERT that
sounds like a pool-health predicate is pointed elsewhere. The line reads
cycle done: N token(s) banked, F above reserve, and in
rotate-reviewer-tokens.sh N is the reviewer pool (real) while F is the
FLEET pool (fresh, computed over ranked ← RECORDS ← ACCOUNTS). Through
the outage the line read 3 token(s) banked, 14 above reserve, and the
ALERT: no account above the … reserve on both axes guard beside it never
fired, because it measures the fleet's accounts. That is a defect in dotclaude,
not here, and it is named rather than fixed in this PR — but it is the reason a
log-match-age on an existing marker could not have carried this signal.
account status written and the detail read the per-account probe results
the cycle writes to ~/.claude/jobs/reviewer-pool/status.json: email, tier
(primary / secondary), live / capped / failed, 5h and 7-day use, when it was
checked — never a token. Gera, 2026-10-22: "show the gh reviewer accounts in
their own table, or a column noting owner". The /agents Systems tab renders
it on this row and as its own table on the ccswitch row. An unreadable pool
leaves the last file in place, so it goes stale rather than reading as empty.
That staleness is account status written's job and the two signals stay
separate on purpose: a fresh file with no live account is a capped pool, a
stale file is a stopped rotator, and one signal covering both could not tell
the reader which it was looking at.
outside-write check ran proves step 0b (reviewer-pool-writer-check.sh,
dotclaude#209) ran this cycle. That step reads the org audit log, for both
created and updated secrets, for reviewer secrets in every org repo that a
login + gh version other than the publisher's wrote. It pages #alerts once per
event (a possible breach; its DESTINATION comment says why, and dotclaude
tests/reviewer-pool-destination.sh pins every reviewer-pool page's channel),
and it reports a personal-account repo
the rotation fed as "coverage unsupported" (exit 4). The signal says the check
ran, not that the secrets are clean: exit 1 (outside write), 3 (could not
tell) and 4 (unsupported) all match.
Both side checks (outside-write and pool-run) are time-boxed at 300s
(dotclaude#271). On 2026-10-05 the pool-run check froze inside a GitHub call,
the pass never ended, and launchd, which will not start a job that is still
running, never fired it again: 10½ days with no scheduled refresh. A timeout
now logs ALERT … TIMED OUT, and the <name> check exit line is kept on every
path, so outside-write check ran still matches.
Appendix
- https://github.com/gera-propflow/dotclaude#readme — the README (no docs declared in SYSTEMS.md yet; add some)
cerebrus (Clara's quality gate)
| What it does | Pipeline — replays a recorded agent turn (or whole thread) through a deterministic three-layer checker and returns pass | repair(span) | hold | block; blocks a propflowai PR when a fixture's verdict regresses against the merge-base |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/cerebrus |
| Deployed at | vendored into propflowai/quality-gate-poc/, pinned by CEREBRUS_VERSION |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → cerebrus (Clara's quality gate) |
Liveness signals
| Signal | How it is read |
|---|---|
| checker scheduled | launchd · co.propflow.cerebrus-gate |
| gate green | log-match-age · ~/.claude/jobs/cerebrus/cerebrus-gate.log · matches GATE OK · stale after 120 min |
| reading fresh | file-age · ~/.claude/jobs/cerebrus/snapshot.json · stale after 120 min |
Detail document: ~/.claude/jobs/cerebrus/snapshot.json (the Systems tab opens this row into it).
Three layers, run in order, higher severity from an earlier layer winning and
never downgrading: L1 diffs the turn against a serialized ground-truth
ledger, L2 checks the tool loop actually backs what was claimed, L3
checks the outbound draft. The harness makes zero LLM calls —
llmCallsUsed is hardcoded 0, and the prototype's "LLM last-resort vote" was
deliberately stripped on port. The one judge lane (judge/invented-policy-lane.ts,
model pinned) is advisory over 5 cases and can never move a verdict.
It never imports application code, and it is vendored into propflowai
rather than depended on — the pin carries a tag, a commit sha, and a
vendored-tree checksum, with check-vendor-sync.sh refusing drift.
The blocking gate is the paired base-vs-head verdict diff, not the corpus
match. The full-corpus run is log-only on purpose: case-11/12/13 and
d8-0e01f3be are disclosed intentional misses, and a gate that failed on them
would be measuring the answer key rather than a regression. The diff fails on
three classes only — a verdict flipping correct→incorrect against
expected.json, a crash that was clean at base, and a case that disappeared.
Because the gate runs entirely on GitHub runners, nothing it does touches
this machine — cerebrus/ledger/ holds a .gitkeep and nothing else, since
the run ledger is written on the runner and dies with it. So liveness here is
supplied by cerebrus-gate (launchd co.propflow.cerebrus-gate, hourly),
which asks GitHub what the gate actually did and writes the answer where the
relay can stat it. GATE OK requires the consumer gate — propflowai's
quality-gate-poc.yml, the one that blocks PRs — to have concluded success
within 14 days, with the engine's own CI not failing. GATE FAIL,
GATE STALE and GATE UNKNOWN each name their reason, and an unreachable
GitHub is UNKNOWN rather than a silent pass.
The staleness window is deliberately loose. The gate is PR-triggered, not scheduled, so a quiet week is normal and must not colour the row; two weeks with no run at all means the workflow was removed, disabled or renamed, which is the silent death worth catching.
⚠️ The signal that looks right and is a trap: CEREBRUS_VERSION's file age.
It is the obvious local artifact and it is a lie — that file's mtime is when
the repo was cloned, not when the gate last ran, so a fresh checkout would
render green on a gate that had been dead for a month. Rendering missing
evidence as health is the exact disease this catalog exists to treat.
Note for searchers: the docs page is titled "Cerberus Burn-In" (with the r),
while the repo, the package and this id are cerebrus. The id here is the
machine key; the docs title is prose.
Appendix — everything linked from this entry
pre-queue type check gate
| What it does | Reflex — before `pr-merge` arms a propflowai PR, the head is type-checked MERGED WITH MAIN'S CURRENT TIP and the arm is refused only on a real red (ADR-0043 is the near copy of the rule, the cure and the fail-open posture; the producer is propflowai's .github/workflows/prequeue-typecheck.yml). |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/pr-merge (typecheck_gate) :: PropFlow-Technologies/propflowai .github/workflows/prequeue-typecheck.yml |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → pre-queue type check gate |
Liveness signals
| Signal | How it is read |
|---|---|
| gate deciding | file-age · ~/.claude/jobs/prequeue-typecheck/gate.json · stale after 4320 min |
Detail document: ~/.claude/jobs/prequeue-typecheck/gate.json (the Systems tab opens this row into it).
What it closes. 2026-10-04/05: 40 merge-queue runs failed CI Checks on TypeScript errors a PR's own CI could not see — the PR was green against an older main, main moved under it in a file the PR never touched, and the merged tree did not type-check; the queue found it one train later and cancelled every PR behind it (#16321 ×29, #15900 ×11). The overlap-only refresh cannot see a break between a main-only file and a PR-only file; a type check of the merged tree can, and it is the same check the queue runs, moved in front of the train.
The signal is gate.json, ONE document pr-merge rewrites atomically on every
gate decision — arm, hold (a refusal) and skipped alike — with at,
repo, pr, head, action, why (local-bin#432). file-age on it says the
gate decided recently; json-file shows what it decided. Three days without a
decision means nobody on this box has armed a propflowai PR through pr-merge
in three days — either the fleet is routing around the tool (guard-bash rule 30
is what makes that hard) or nothing merged, and both are worth a look. The
merge-actor ledger (receipts.jsonl) also carries the decision per arm, but it
is JSONL with no leading timestamp, which the catalog's log-match-age cannot
read — the first draft of this entry pointed there and could never have gone
green (agentflow#294 review).
⚠️ The instrument that looks right and is a trap: the prequeue-typecheck
status on a PR head. A success there says the merged tree type-checked AT
THAT MOMENT; main moves every ~2.5 min, and the gate deliberately checks once
per head (one check, one wait; re-checking on every main move is a loop at
main's cadence). A green status is therefore evidence about a window, not a
guarantee at merge time — the residual is stated in the ADR, and the queue's
own run remains the last word.
⚠️ And the queue sees one thing this gate cannot. The queue's Build type check is tsc over the BUILT tree (.next/types included); this gate runs tsgo
over the source tree and cannot see a Next route-type error. None of the three
episodes were of that kind; the nightly tsgo/tsc parity lane in propflowai's
ci.yml is the instrument for a checker divergence.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
pre-arm gate
| What it does | Reflex — before `pr-merge` arms a propflowai PR, ONE read-only GraphQL read of the PR's own records decides whether it may be armed at all, and the arm is refused BOTH on a named condition and on evidence that cannot be read (ADR-0061 carries the polarity rule and why it is the opposite of its sibling's). |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/pr-merge (armgate_gate) :: ~/.local/bin/armgate/armgate.py + armgate/gate.graphql |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → pre-arm gate |
Liveness signals
| Signal | How it is read |
|---|---|
| gate deciding | file-age · ~/.claude/jobs/armgate/gate.json · stale after 4320 min |
Detail document: ~/.claude/jobs/armgate/gate.json (the Systems tab opens this row into it).
⚠️ status is proposed, and the signal path does not exist yet. The gate is wired in an OPEN,
held pull request; nothing on this box runs it until #615 lands and is pulled onto the mini. So
~/.claude/jobs/armgate/gate.json is absent today and file-age on it reads dark — which is
the honest reading, not a defect to paper over. Flipping this entry to active is part of landing
it, and a status: active entry whose signal has never been written once would be exactly the
"configured, therefore installed" claim ADR-0004 exists to refuse.
What it closes, and each check was paid for by an incident (armgate/README.md names the
incident beside the check): a hold-for-review label or a draft armed anyway — #6719,
2026-09-01, shipped its unread email wording to production because auto-merge was armed on a held
PR and a later worker cleared the last objection; a merge-queue count that read NULL treated as
zero — the line hit 60 on 2026-10-03, and every entry is a full CI run on top of main; an older
red same-name review run beside a green one; an absent required review check read as a pass,
because absence is not a verdict; and a CHANGES_REQUESTED that is the newest record at the head.
That last one carries the measurement that forced a zero tolerance: GitHub rewrites an
already-submitted review and moves its commit pin forward, so an old objection reads as live
against code the reviewer never saw — 111 of 111 moved records carried updatedAt != submittedAt,
0 of 183 un-moved ones did, 294/294 concordance, smallest real gap 12 seconds. Clearing such a
record therefore needs BOTH markers, and an unreadable one keeps holding.
The signal is gate.json, one document pr-merge rewrites atomically on every decision —
arm, hold, could-not-look and skipped alike — carrying gate: "armgate". That gate
field is load-bearing: there are two gate snapshots beside each other now, and a catalog signal
that cannot tell them apart reads one gate's liveness as the other's. Its sibling's document at
~/.claude/jobs/prequeue-typecheck/gate.json is untouched by an armgate decision, which the caller
suite asserts.
⚠️ This gate fails CLOSED while its sibling fails OPEN, and that is deliberate — see ADR-0061.
ADR-0043 states its fail-open premise inline, "the cost of a wrong arm is one failed train, which
is today", and the queue catches that one. This gate's wrong arm is #6719 or a live objection
merged, and nothing downstream catches either. The second door is --skip-armgate <reason>,
recorded in the ledger and in this snapshot.
⚠️ The residual nobody should assume away: a refusal here writes no
prequeue-typecheck/refused marker, so guard-bash rule 30 — whose marker is written from one
call site inside typecheck_gate, whose deny prose says "does NOT type-check", and whose cure
names --skip-typecheck — only nudges after an armgate refusal instead of denying the obvious
workaround. Tracked as its own board row rather than fixed here: a guard is not edited on a
second-hand reading.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
pre-queue status alarm
| What it does | Alarm, detection only. It names every open propflowai PR that is armed (auto-merge on, or in the merge queue) and whose current head has no `prequeue-typecheck` verdict: the status is absent, or still pending more than 600 s after max(arm, head commit). For each one it gives the head SHA, the arming actor (newest of autoMergeRequest / AutoMergeEnabledEvent / AddedToMergeQueueEvent), the time since the arm and the status. Each tick makes one GraphQL query as the App (cost 2), cached for 5 minutes. It writes ~/.claude/jobs/prequeue-typecheck/status-alarm.json and appends a stamped line to status-alarm.log. It posts to #agent-smith through the Smith bot when a new PR is flagged, at most every 6 h while any stays flagged, and once on recovery. It never labels, comments, disarms, re-dispatches or touches a ruleset. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/prequeue-status-alarm, spawned detached by ci-factory-mq's 15-minute --health reconcile (LaunchAgent co.propflow.ci-factory-mq) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → pre-queue status alarm |
Liveness signals
| Signal | How it is read |
|---|---|
| alarm measuring (a fresh GitHub read, OK or BAD) | log-match-age · ~/.claude/jobs/prequeue-typecheck/status-alarm.log · matches src=github · stale after 60 min |
| every armed PR has its pre-queue check | log-match-age · ~/.claude/jobs/prequeue-typecheck/status-alarm.log · matches prequeue-status: OK · stale after 60 min |
Detail document: ~/.claude/jobs/prequeue-typecheck/status-alarm.json (the Systems tab opens this row into it).
Why it exists. propflowai#17735 (2026-10-08) made the producer stamp
prequeue-typecheck on every arm, from any machine. The prequeue-typecheck entry
above has only one signal, the age of this box's gate.json, and that cannot see an
arm made elsewhere whose status never arrived. Possible causes are an Actions outage,
a dropped webhook, a dispatch with no run behind it, or an arm that routes around the
gate. Once the status is required, such a PR waits forever with nothing saying so.
The two signals split on purpose. src=github is liveness: it appears on any line
written after a fresh read of GitHub, whether the verdict is OK or BAD. A cache hit says
src=cache, and a failed read says UNKNOWN. prequeue-status: OK is the alarm itself:
it goes stale exactly while some armed PR is missing its check. On the first run it reads
down: the 6 PRs flagged at 06:20Z (five armed by fede-propflow, one by gera-propflow) were
all armed before #17735 merged, so they predate the trigger. That reading is the honest
one, not a broken signal.
The grace is measured. Over 108 completed runs from 2026-10-06, created to completed: median 123 s, p90 144 s, p99 183 s, max 187 s. A 600 s grace would flag none of them. A queued run has no status at all until it starts, so "absent" and "pending" are one rule.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
pr-autoqueue (external enqueue loop)
| What it does | Every 300 s (LaunchDaemon), for each repo listed in ~/.claude/jobs/pr-autoqueue/repos.txt (one owner/repo per line; today propflowai, agent-harness and tenio (since 2026-11-04); missing/empty/malformed = `tick UNKNOWN`, nothing queued; `pr-autoqueue repos` lists merge-queue repos NOT configured, and adding a line is the activation), reads every open PR as the App and runs `pr-merge <url> --class guard --no-wait` on each one that is 🟢 on its CURRENT head, based on main, not a draft, not `hold-for-review`, not `[HOLD]`-titled, not Dependabot, not conflicting, not ejected from the queue at this head, with no standing CHANGES_REQUESTED, NOT already armed or queued (it never touches someone else's arm), and with no required check affirmatively RED on a fresh re-read of the head. The 🟢 is the review bot's newest decisive formal review on the head; where there is none (agent-harness: bodyless COMMENTED reviews, and its required `review` check is green on 🔴 heads) it is read from the bot's finished comment, counting only the reviewer's verdict HEADER on the first report line, anchored to the head through the linked Actions run, with every review workflow's newest run on the head finished and none 🔴/🟡/adversarial HIGH. A 🟢 older than 24h (PR_AUTOQUEUE_MAX_GREEN_AGE_H) is logged STALE-GREEN and not queued. Per repo: queue cap 29 (queue entries + armed-not-yet-queued), at most 5 per tick oldest first, failed-heads keys owner/repo#n, refusal markers, log prefix `<owner/repo> #N WORD`, status.json `repos` section, and a kill switch `STOP.<owner>--<repo>` beside the global `~/.claude/jobs/pr-autoqueue/STOP`. The type-check gate and the receipt stay in `pr-merge` (ADR-0043); a head the gate refused, or one pr-merge FAILED on, is skipped until it changes. One log line per decision, and one global `tick ok mode=live repos=N` line per pass, written only when every configured repo was read (one unreadable repo makes it `tick PARTIAL`; the others still run). Since local-bin#663 it also keeps a STUCK LEDGER (~/.claude/jobs/pr-autoqueue/stuck.json): a PR it refuses for STUCK_TICKS consecutive ticks at ONE unchanged head, whose head commit was younger than STUCK_MAX_HEAD_AGE_S when the latch opened, is announced ONCE as `#N STUCK` (and once more as `#N UNSTUCK` on the cure, never per tick), and every pass writes one `stuck census ok|BAD|UNKNOWN` line plus a `stuck` block in status.json. That census is the second signal below; `tick ok` cannot carry a verdict. It never uses --admin, never disarms, labels or comments itself, and never touches a ruleset or branch protection. SINCE 2026-10-27 (local-bin#679/#681/#682/#683/#684, Gera approved) every LIVE tick also runs two no-AI sidecars after its status is written, each with its own log and verdict line: `pr-reconcile` makes every open gera-propflow PR in PropFlow-Technologies carry a live review-sub subscription to a LIVE owner (worktree holding the head branch, else the session that ran pr-merge on it this week, else ~/.claude/jobs/pr-autoqueue/main-session for PRs under 72h; one `reconcile ATTACHED` line per attach, `reconcile UNOWNED` once per PR; ~/.claude/jobs/pr-autoqueue/reconcile.log), and `pr-stuck-wake` wakes the OWNER of an armed or subscribed PR that has sat 30 min at one head with a CHANGES_REQUESTED review, a failed required check with nothing running, or conflicts with main (behind main is not stuck): once per episode and once on the cure, typed into the owner's tmux session the way the listener types its wakes, falling back to pr-reconcile's owner when the owner has ended and, only with no live owner at all, ONE post to the mini-alerts destination (stuck-wake.log). A tick under test (PR_AUTOQUEUE_DIR repointed) runs neither. pr-merge exit 9 (REFUSED-NO-OWNER here, unpinned) refuses to arm a gera-propflow PropFlow-Technologies PR with no live subscription. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | ~/.local/bin/pr-autoqueue, LaunchDaemon co.propflow.pr-autoqueue (system domain, StartInterval 300; launchd/co.propflow.pr-autoqueue.plist) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → pr-autoqueue (external enqueue loop) |
Liveness signals
| Signal | How it is read |
|---|---|
| a tick read GitHub and decided | log-match-age · ~/.claude/jobs/pr-autoqueue/autoqueue.log · matches tick ok mode=live · stale after 30 min |
| no PR has been refused for an hour at one unchanged head | log-match-age · ~/.claude/jobs/pr-autoqueue/autoqueue.log · matches stuck census ok mode=live · stale after 30 min |
| every open PR from Gera's machines was checked for a live owner | log-match-age · ~/.claude/jobs/pr-autoqueue/reconcile.log · matches reconcile ok mode=live · stale after 30 min |
| stuck PRs were checked and their owners woken | log-match-age · ~/.claude/jobs/pr-autoqueue/stuck-wake.log · matches stuck-wake ok mode=live · stale after 30 min |
Detail document: ~/.claude/jobs/pr-autoqueue/status.json (the Systems tab opens this row into it).
Why it exists. Fede's retirement of auto-merge-all.yml (propflowai #14580..#14737,
2026-10-01) moved every rule that workflow enforced into required checks and assumed "an
external loop enqueues approved + green PRs". Nothing did, so from 10-01 to 10-15 a PR
reached the merge queue only when a session remembered to run pr-merge. Gera chose to
build the loop rather than revive the workflow (2026-10-15).
No launchd signal, on purpose. The job is a system-domain LaunchDaemon, and the relay's
launchd evaluator runs launchctl list in the user domain, which cannot see it (ADR-0055).
It would read a permanent false DOWN. The log line is the liveness, as for doctrine-pull.
What the first signal proves. tick ok mode=live is written only after a full read
of GitHub (open PRs + required check names) and a decision on every PR. A read failure
writes tick UNKNOWN, and the kill switch writes tick stopped, so both go stale on
this row instead of reading healthy. A dry run writes mode=dry-run and does not count.
⚠️ AND WHAT IT CANNOT PROVE — why the second signal exists. tick ok measures that the
loop RAN and never what it DECIDED, so it cannot go red while the loop is alive, however
bad the news is. Measured: agent-harness#1384, the fix for the shared-checkout wedge, drew
222 SKIP lines from this loop between 2026-10-24 13:28:37 and 2026-10-25 08:35:53 local,
188 of them conflicts with main — straight through the wedge it was the fix for and through
the hand recovery — and this row stayed green the entire time, because every one of those ticks
did read GitHub and did decide. The same log says conflicts with main 10,233 times, so
a signal on the state itself would be permanently red, which is the same defect one layer out.
stuck census ok mode=live is the verdict line instead: pr-autoqueue latches a refused PR
on (PR, head sha) and writes stuck census ok only when no latch has stood for
STUCK_TICKS consecutive ticks at one unchanged head, every configured repo was readable,
and every refused PR's head commit was datable — otherwise BAD or UNKNOWN, neither of which
the pattern matches, so the row goes stale and reddens. A PR merely BEHIND main never enters
it: behind reads mergeable: MERGEABLE at any distance behind, is armed as usual, and the
queue tests the merge result; the latch is built on CONFLICTING and reports only its
persistence. Entry is gated on the head commit's age — STUCK_MAX_HEAD_AGE_S, its own knob, which
merely DEFAULTS to the same 24 h as MAX_GREEN_AGE_H rather than being derived from it —
which is what keeps it a signal: without that gate the same replay fires 38 times instead of
5, because 48 abandoned agent-harness PRs with heads 32 h to 41 days old latched the
moment that repo entered repos.txt. Both numbers live in the program, not here —
STUCK_TICKS and STUCK_MAX_HEAD_AGE_S in ~/.local/bin/pr-autoqueue, with a dated snapshot
in this entry's _signals_note. Derivation, the refuted alternative entry rule and the sabotage
controls: PropFlow-Technologies/local-bin#663.
The cutover is done, and it was the ordering risk. A signal declared before its writer
exists has nothing to match, so this row would have read DOWN for a reason that is not a fault
— and a row red for no reason is how people learn to ignore the row, which is the condition
this whole change exists to end. local-bin#663 is merged, doctrine-pull has fast-forwarded
~/.local/bin to it, and at 2026-10-25 16:05:51 local the log carries stuck census ok mode=live threshold=12 prs=0 latched=1 with stuck.json written, so this row reads from a
line that exists. The caveat is carried in-band in _signals_note as well as here, because the
next reader of a catalog is a renderer or a script and neither reads paragraphs
(ci-factory-mq set that precedent for its auth rows). The pattern is a substring match and
this log quotes PR TITLES back in its decision lines, so a PR titled with this exact phrase
could in principle false-green it; that is the known limit of log-match-age, not a reason to
make the marker vaguer.
Appendix
- https://github.com/PropFlow-Technologies/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
merge-evidence gate
| What it does | Reflex — refuses to arm GitHub auto-merge on a propflowai PR unless every required context has concluded AND produced evidence about the SHA being armed, so a green that measured nothing cannot merge unattended |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | .github/workflows/auto-merge-all.yml rule 6 :: scripts/ci/decide-merge-evidence.sh |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → merge-evidence gate |
Liveness signals
| Signal | How it is read |
|---|---|
| gate deciding | log-match-age · ~/.claude/jobs/merge-evidence/merge-evidence-gate.log · matches EVIDENCE-GATE OK · stale after 20160 min |
Detail document: ~/.claude/jobs/merge-evidence/snapshot.json (the Systems tab opens this row into it).
The hazard it closes. Auto-merge merges on green, and several of propflowai's
greens are statements about a SELECTION or a PATH FILTER rather than about
anything that ran: Unit Tests = success with the affected selection skipped
for exceeding AFFECTED_CAP; ci-docs.yml's unconditional stub — it forges
FOUR required names in one loop (Type Check, Unit Tests, Build, Lint),
titled "docs-only change — skipped" — landing on a diff that is not docs-only; a required context that only ever
skipped (protection passes those); and a PR into a feature-branch base,
where "all checks passed" is true over an EMPTY required set because
auto-merge-all.yml has no base filter and ci.yml only runs for
branches: [main].
It is a gate, not a brake, and the distinction is the design. The rejected
version wrote a hold-for-review label from a periodic observer that notices a
vacuous green — a label applied by a poll can land AFTER the merge has already
happened. Eligibility is instead evaluated at ARMING time against the SHA being
armed, and arm() hands that SHA to --match-head-commit, which GitHub
validates at merge time.
⚠️ THE STUB AND THE REAL LANE SHARE A REQUIRED NAME, AND NOTHING AT READ TIME
TELLS THEM APART. 2026-09-11, dated and costed: a reviewer read Lint: success titled "docs-only change — skipped" on a mixed docs+code PR, concluded
the rename ratchet had never run, and held a merge that was unblocking the whole
repository. The real Lint HAD run, for 552s; the 17-second stub posts under
the same name and arrives first. An hour lost — and note the direction: that was
a FALSE HOLD by a human whose only available discriminator was DURATION. The
gate answers it by asking WHICH PRODUCER posted, which is the same question in
every direction: stub+real → eligible, stub alone on a code diff → held, stub
alone on a docs-only diff → eligible.
⚠️ AND BRANCH PROTECTION REPORTS THE STUB AS THE ANSWER. Observed twice on
this gate's own pull request, on consecutive commits: all six required contexts
reading pass while ci.yml's real lane had not reported — the second time
over a lane that was actually FAILING. Any instrument that asks
gh pr checks --required inherits this; ask the producing lane
(external_id == ci-yml-unit-tests) instead.
⚠️ IT NEVER READS A DURATION, and the catalog should not let anyone "fix"
that. Zero duration alone is not proof of invalid testing; MISSING REQUIRED
EVIDENCE is the reason to hold. propflowai's own data is anti-correlated: on
#7583 the vacuous Unit Tests green ran 3m45s while a genuine review context
ran 0s (GitHub rounds to whole seconds). A duration gate blocks honest work,
gets routed around, and leaves the repo worse than before. Tests scan the
shipped source and fail if any timestamp field is ever read.
⚠️ THE SIGNAL THAT LOOKS RIGHT AND IS A TRAP: "the workflow exists", or "the
auto-merge workflow ran." Both stay true of a workflow whose evidence step has
been deleted, and of one whose script dies exit 127 before deciding — which is
exactly how a chokepoint in this repo failed once before. So merge-evidence-gate
(~/.local/bin/merge-evidence-gate) reads the DECISION ITSELF: it walks recent
successful auto-merge-all.yml runs and greps their logs for the
decide-merge-evidence: state=… line, which carries sub-second timestamps.
Most runs exit early ("No open PR for this event"), so it scans up to 25 rather
than only the newest — reading one run would render a working gate as gone.
⚠️ IT DOES NOT GRADE THE VERDICT, deliberately. state=missing is the gate
WORKING: it held a PR that had no evidence. A checker that went red on
missing would be standing pressure to loosen the gate — the ratchet this
whole effort exists to resist. What is measured is that the gate DECIDED; the
ok/pending/missing mix is reported as data on every sample.
The window is 14 days and is deliberately loose. The gate is PR-triggered, not scheduled, so a quiet day is normal and must not colour the row. Two weeks with no decision at all means it was removed, disabled or renamed.
Both controls are observed, which is the point. Its first samples
(2026-09-10 23:39 and 23:49, before the rule reached main) read EVIDENCE-GATE GONE; at 23:50, once a real guard run had logged decide-merge-evidence: state=missing, it read EVIDENCE-GATE OK decided@0.0d mix=missing=1. A signal
that cannot say NO is not a signal, and one that can only say NO is not one
either — this one has said both, each for the right reason.
⚠️ THE WINDOW IS MEASURED IN RUNS, AND THE RUN RATE IS ENORMOUS — the second
GONE above was a FALSE one. auto-merge-all.yml fires on every PR event in a
very busy repo: ~30 successful runs in seven minutes, measured. The checker's
first draft scanned 25 and reported GONE against a gate that had decided
correctly eight minutes earlier, because that run was already 31 back. A window
measured in RUNS inherits the watched workflow's run rate; at this one's rate,
25 runs is fifteen minutes of history and the signal becomes a generator that
mostly says dead. It scans 120 now (about an hour of traffic) and stops at the
first decision, so the healthy path stays cheap and only the genuinely-broken
path pays the full scan. Any future signal over a high-traffic workflow inherits
this trap.
⚠️ NOT SCHEDULED YET, and that is a deliberate stop. The plist is written
(co.propflow.merge-evidence-gate, hourly, mirroring co.propflow.cerebrus-gate)
but NOT loaded: installing a cron schedule is inside the founder's risky-change
carve-out. Until someone runs launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/co.propflow.merge-evidence-gate.plist, this row ages out
to UNKNOWN after 14 days rather than rendering healthy — which is the correct
failure direction and is why the gap is written here instead of papered over.
Appendix
- https://github.com/PropFlow-Technologies/propflowai#readme — the README (no docs declared in SYSTEMS.md yet; add some)
local-bin land-on-green lander
| What it does | Reflex — merges a same-repo local-bin PR once its Review Gate verdict and run-all are green on the exact head, from the DEFAULT branch's copy of the workflow, so a PR cannot rewrite the guard that judges it |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | GitHub Actions :: .github/workflows/land-on-green.yml (workflow_run on 'Review Gate' and 'run-all' completed) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → local-bin land-on-green lander |
Liveness signals
| Signal | How it is read |
|---|---|
| a merge reached the mini's local-bin checkout | log-match-age · ~/.claude/jobs/doctrine-pull/doctrine-pull.log · matches state=ADVANCED · stale after 2880 min |
What it decides, in order (all from the API, never from the PR's code, the
triggering run's outputs or its pull_requests payload):
- Only a same-repo
pull_requestrun is a candidate, and the head must be a 40-hex SHA. - The PR is resolved from
head_sha, across every page of results. Zero or more than one open match is a refusal. hold-for-reviewand draft are holds, and so is a label or draft state it cannot read.- Any change under
.github/(or toCLAUDE.md) is held for a human. This is computed BEFORE any status is read, because such a PR ran its ownreview.ymland could have written that status itself. review(written by the workflow token) andrun-all (hosted-safe suites)must both be green, by name.- The merge passes
sha=<head_sha>, so GitHub refuses it if the head moved after the review.
Why it is a separate workflow. The job it replaced sat at the bottom of
review.yml and ran from the PR's OWN copy of that file. A same-repo PR could
delete its own .github/ hold and then be merged by the guard it had just
rewritten. workflow_run always runs the default branch's copy. There is one
lander. A second one beside it is the two-merge-owners bug (ADR-0030).
Two triggers, on purpose. Woken by the Review Gate alone, it would find run-all still running and decline every PR, which is exactly what the in-run job did. It recomputes every gate whichever run woke it. The later of the two lands the PR; the earlier one declines.
Its own test is tests/land-on-green-refuses-what-a-human-owns.sh in local-bin,
which runs the exact step against a stubbed gh.
Appendix
- https://github.com/PropFlow-Technologies/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
decision blocks
| What it does | Carries a question from a blocked session to Gera and the answer back, then tracks whether the answer was APPLIED — `blocked raise` files it and publishes a decisions-page link, one durable Temporal driver per drivable row polls the single answer store and wakes the lane that owns the work (the raiser, or whoever holds its task now), the 600s sweep drives OPEN rows AND answered-but-unapplied ones and publishes the unapplied backlog as counts, and `blocked outstanding` names every decided row with no terminal application — including the ones blocked on a capability no agent has |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | ~/.local/bin/blocked (raise · check · resolve · apply · executed · verified · outstanding); agent_smith.decision_block_cli sweep (launchd, 600s); agent_smith workflows/decision_block.py (Temporal) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → decision blocks |
Liveness signals
| Signal | How it is read |
|---|---|
| sweep scheduled | launchd · co.propflow.decision-block-sweep |
| sweep completing | file-age · ~/.claude/jobs/decision-drivers.json · stale after 30 min |
Detail document: ~/.claude/jobs/decision-drivers.json (the Systems tab opens this row into it).
Why this entry exists at all (2026-09-07). It did not, and the omission was
load-bearing rather than tidy: this is the path every answer Gera gives travels,
it has a launchd job and a durable workflow per question, and nothing in the
catalog watched it. On 2026-09-07 six decisions he had answered on the
decisions page — each row carrying his verbatim choice — sat between one and six
days with their raisers' tasks still awaiting-human, the state whose whole
promise (the Supervisor's tasks.ts, since deleted) was that "when the answer lands there is
still somebody whose job is to execute it". Nobody was, and no signal anywhere
said so. A system absent from this file is a system nobody is watching; that is
the rule this file opens with, and this is what it costs.
Why the liveness signal is a FILE AGE and not a log match. A completed sweep
prints one of two terminal lines — sweep: N drivable, M skipped, or no open blocks in the ledger — nothing to drive — and they share no substring. The
relay matches log patterns as SUBSTRINGS, not regexes (the HOST (OK|PRESSURE)
lesson in the host entry above, which rendered down for a sampler writing
every five minutes), so ANY pattern keyed on one of those two lines reads down
in a healthy state: an empty ledger is the good outcome and would have been
the one reported as broken. decision-drivers.json is written on every
completed non-dry path — the empty ledger, the all-undrivable case, and the
normal one — deliberately, so that "nothing is being driven" is stated freshly
rather than left to the previous sweep's file. That makes its mtime the honest
proof the sweep ran, whatever it found.
⚠️ THIS ENTRY SAID "IT READS OPEN BLOCKS ONLY, CORRECTLY" AND THAT WAS FALSE
FROM THE DAY ADR-0013 SHIPPED. The prose above and the job string both
described an OPEN-only sweep; the code has swept answered-but-unexecuted rows
since ADR-0013 (open_block_records(..., unexecuted=True)). Corrected
2026-09-10 along with ADR-0022. Recorded rather than quietly fixed because it is
the same defect class this file exists to catch — a description that reports
green on behaviour that has moved underneath it — and catalog prose rots
faster than anything else here: the claim was contradicted by a merge in the
same repo, eleven days after it was written, and nothing failed.
What this system does NOT do, stated so nobody assumes it. Closing a block
is not executing its answer, and tracking an answer is not applying one.
The sweep drives an answered row only while the lane that owns it is alive
(ADR-0013's liveness bound, ADR-0022's task-keyed successor), so an answer whose
lane is dead is recorded, durable, correct and unapplied — named by blocked outstanding for a person, never executed by anything here. It executes nothing
itself: it polls, it wakes, and it records what somebody else says happened.
Three of those non-facts are worth naming individually, because each one used to be invisible:
- A row can be DECIDED and its application BLOCKED on a capability — a Slack
kickoff to Smith, where every agent route is structurally dead. The sweep will
not drive it (no wake fixes a missing capability) and does not hide it: it is
skipped by name and counted in the manifest under
unexecuted.needs. It is a queue for a person's hands, and it is not an unanswered question. - An APPLIED row is not a VERIFIED one. A merged PR stays merged forever while the behaviour it added is reverted underneath it, so an application record whose evidence names no revision reads "verified against no named revision" — a weaker claim, not a passing one.
- Historical rows carry no
taskstamp and are not retro-fitted, so the ~176 answered-unapplied rows on this box became countable and triageable the day ADR-0022 landed, but only rows raised after it are automatically inherited by a successor.
Why the DETAIL file now carries the backlog, and why it is omitted rather than
zeroed. decision-drivers.json grew an unexecuted block — total, counts by
application state, the capabilities decided work is stuck behind, and the age of
the oldest row. That last field is the one that matters: a total can sit flat
while every row in it ages, so "how long has the oldest answer been waiting" is
the number that says a backlog is growing. The counts are taken WITHOUT the
liveness narrowing the drive list uses, because a row whose lane is gone is the
most outstanding row there is — narrow the count the same way and it reads zero
exactly when things are worst. And a sweep that could not count OMITS the field
instead of writing 0: a zero nobody measured is this file's own
frozen-job-reports-green failure, one field down.
Asking Smith is a deferral, not an answer (2026-10-08, ADR-0048). An
open decision on /agents carries an "Ask Agent Smith" control. It copies a
message that opens with Decision <id> and opens Gera's Slack DM with Smith.
It writes nothing: the row stays open, the sweep and the per-row driver keep
treating it as unanswered, and the raiser keeps waiting. The answer still lands
on the raiser's row, either from Gera on /agents or from Smith running blocked resolve <id> --sid <raiser> --answered --choice '<their words>' --by <decider> (the
--sid is required, because the row lives in the raiser's ledger). From there
this path wakes the raiser as it does for any answer. Nothing marks the row
"asked". waiting_on means a waiter queued behind another session's block,
and reusing it would drop the question out of every sweep.
Appendix — everything linked from this entry
- docs/adr/0013-answered-is-not-executed-and-the-sweep-chases-both.md
- docs/adr/0022-a-decision-carries-three-facts-and-application-binds-to-the-row.md
- docs/adr/0048-asking-smith-about-a-decision-is-a-deferral-and-the-answer-still-lands-on-the-row.md
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README
decision ladder
| What it does | Answers a fork a working session cannot settle, without waking anybody — per ADR-0019 Fable and Astra are asked INDEPENDENTLY on one evidence snapshot and agreement is the decision, a disagreement gives Fable one reconsideration whose actionable verdict wins; every call is `fable-decide`, so every call is two options-reversed passes carrying the money/commitment gate, and within a call Fable still falls through to Astra when Fable cannot be REACHED (never when Fable was reached and declined); the answering model is stamped on the receipt, and a 300s canary writes `ladder OK` only while both model rungs are reachable |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/fable-decide (rung 1 `claude -p --model claude-fable-5`, rung 2 `codex exec -m gpt-6-astra`); receipts ~/.claude/jobs/fable/<id>.json; ~/.local/bin/decision-ladder-canary (launchd co.propflow.decision-ladder-canary, 300s) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → decision ladder |
Liveness signals
| Signal | How it is read |
|---|---|
| canary scheduled | launchd · co.propflow.decision-ladder-canary |
| both model rungs reachable | log-match-age · ~/.claude/logs/decision-ladder-canary.log · matches ladder OK · stale after 15 min |
Detail document: ~/.claude/jobs/decision-ladder.json (the Systems tab opens this row into it).
Why this entry exists (2026-09-10). It did not, and this is the rung between
a stuck session and Gera's decisions page — the one that decides whether he is
woken at all. On 2026-09-10 it lost its only model rung for ninety minutes and
nothing said so: six of seven seats weekly-walled, the seventh session-capped
until 20:40Z, five questions on PR #7618 reported "pending Fable" while
codex exec -m gpt-6-astra answered in 7 seconds when finally asked.
local-bin#107 made the cap a wall (exit 4, stop retrying); local-bin#109
added the Astra rung, so the ladder degrades Fable → Astra → human instead
of vanishing; local-bin#110 is the canary that lets this row exist.
decision-blocks (above) catalogs the path an answer travels after a human
is asked; this is the path that decides whether to ask one.
Why the signal is a log-match on the literal ladder OK, and not a file age.
fable-decide is on demand — a quiet day writes no receipts — so a file-age
over jobs/fable/ would render a healthy quiet day as degraded, the exact trap
the decision-blocks entry documents from the other direction. The canary is
the thing that CAN fail: it runs on a clock, reads ccswitch's snapshot
(classified by fable-decide's own pick_seat(), so it cannot disagree with
the tool it watches) and the codex access token's exp claim, calls no model,
and writes one line every run. DEGRADED and DOWN lines are written too, so a
reader sees what is wrong — but they do not match ladder OK, so this row
goes stale precisely when a rung is missing (900s = three missed ticks).
The launchd signal is the other fact: the canary exits 0 whatever it finds,
so "the canary crashed" and "the ladder is down" are different colours, and
they need different people.
The one thing that must never be "simplified" here. Could not be reached
and was reached and declined must not share an exit path. A capped seat and a
genuine resolvable: false are opposite facts, and a fall-through keyed on
"non-zero exit" would hand a question a model said only a human may settle to a
second model, which could return it settled. The discriminator is one pure
function — did any pass return a parseable verdict object? — and
tests/control-20-fable-decide.sh §G asserts every refusal path with a tripwire
Astra stub that WOULD resolve, so a swallowed refusal cannot pass as green.
Appendix — everything linked from this entry
puller — the Driver's feed, and the decider
| What it does | Reads the Driver's PRs, main's CI and the decision queue every 300s and writes ONE delta digest, so the Driver judges instead of polling — and DECIDES un-ruled blocks by running the ADR-0019 ladder, then per ADR-0020 hands the ruling BACK to the blocked raiser in one message naming the choice, the receipt and the model, without ever resolving one itself — and, per ADR-0030, lists every IDLE PR (nothing running, nothing moved 45m+) under the session that owns it |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/puller (launchd co.propflow.puller); ledger ~/.claude/jobs/puller/routed.json; spend ~/.claude/jobs/puller/spend.jsonl |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → puller — the Driver's feed, and the decider |
Liveness signals
| Signal | How it is read |
|---|---|
| scheduled | launchd · co.propflow.puller |
| reading the fleet | log-match-age · ~/.claude/logs/puller.log · matches puller OK · stale after 15 min |
| digest fresh | file-age · ~/.claude/jobs/puller/digest.md · stale after 15 min |
| rulings reaching raisers | log-match-age · ~/.claude/logs/puller.log · matches puller DELIVERED · stale after 4320 min |
| idle-PR detector seeing | log-match-age · ~/.claude/logs/puller.log · matches puller IDLE · stale after 15 min |
| orphan mark seeing | log-match-age · ~/.claude/logs/puller.log · matches orphans= · stale after 15 min |
| reading GitHub as the propflow-agent-smith App (not the shared user account) | log-match-age · ~/.claude/logs/puller.log · matches auth=app · stale after 15 min |
Why it exists. Gera, 2026-09-10: "we have to have something feeding you."
The Driver swept PR states, check conclusions, review verdicts and the
decision queue by hand on every wake, mostly to learn nothing had changed. In
one evening that cost a PR merged out from under it, a regenerating alarm it
lost one withdrawal at a time, and two problems Gera saw before the Driver
did. driver/SKILL.md §1 already said "watching is a dumb process's
job"; no such process existed. ADR-0017
is the rule; local-bin#113
specifies the rung and #114
builds it.
It reads, decides, and returns. It never acts on a ruling — no merge, no
resolve, no approve, no label, no draft toggle, no production write, no registry
write, and it never answers a decision page or writes state on a block. One
writer per artifact (ADR-0003); the only artifacts it writes are its own. It is
not a nudge watcher (launchd, no loop, no GATE, never in the nudge
roster — §0b would be right to kill a second looper).
The one thing it sends is a single message to a single blocked raiser,
carrying the ruling this rung obtained for that raiser's own question
(ADR-0020). That is the decisions wake the blocked contract always
promised, restored on a path that works — an answer clicked on the /agents
Decisions tab wakes the raiser zero times. It is not a delta ping, it
never reaches the Driver, and §1's ~30-firings cap is untouched.
Why six signals, and why they are not redundant. They answer six different questions and the row takes the worst of them:
| signal | the fact it carries | what its absence means |
|---|---|---|
launchd |
the job is loaded and not crash-looping | nobody bootstrapped it, or it is failing to start |
digest fresh |
a tick RAN and produced a file | the process is dead or wedged |
puller OK |
a tick ran and could read every source | it is running blind — GitHub refused, or the decision queue is gone |
puller DELIVERED |
a ruling actually reached a blocked raiser | the return hop is broken, or nothing has needed deciding |
puller IDLE |
the idle-PR detector (digest §5) could look: PR listing read, owners resolved | the detector is blind — see below |
orphans= |
the ORPHANED mark could look: owner liveness was read this tick | every row reads not-orphaned because nobody could check, not because every owner is alive |
⚠️ ORPHANED is a MARK, not an act (ADR-0030
amendment 2026-09-25, local-bin #264/#267). §5 marks an owner group ORPHANED
when liveness batch reads the owning session DEAD, and writes the rows to
last.json orphans. The rung that closed ORPHANED rows idle ≥ 7d,
cleaner --prs, was retired 2026-10-10 having closed none in 305 runs; the Driver check-in (local-bin #265) hands the < 7d ones to
the live Driver whose area they are in. The mark needs positive evidence: a
failed or unknown probe is never ORPHANED, and a row with no owner stamp is
marked adoptable but is never auto-closed. The orphan mark seeing signal
matches orphans=, which the tick writes only when it read liveness; a blind
probe writes orphans-UNREAD instead. The match is a substring, not a regex
(relay/src/systems/catalog.ts).
⚠️ puller IDLE FAILS WHEN THE DETECTOR BREAKS, NOT WHEN PRs GO IDLE
(ADR-0030,
ADR-0009). It is written
every tick, n=0 included — "nothing idle" is a finding, so a clean board keeps
it green. It is replaced by puller IDLE-BLIND why=…, which the pattern (note
the trailing space) does not match, when the PR listing is unread, the owner
scan cannot run, or the scan resolves zero owners across open PRs. That last
case is the one a naive signal would miss: owners come from the "gitOperation": {"pr":…,"action":"created"} stamp Claude Code writes into the creating
session's transcript, and if that harness-internal shape ever changes the scan
still exits 0 — it just finds nothing, and every row would read owned-by-nobody.
Keyed on puller OK alone, all three would stay green.
⚠️ puller DELIVERED CARRIES A THREE-DAY WINDOW, NOT FIFTEEN MINUTES, AND
THAT IS THE HONEST SHAPE. Deciding is demand-driven — a quiet week raises no
blocks and delivers nothing, and a signal that went red for a quiet week would
train everyone to ignore it (ADR-0009 cuts both ways: a signal that fails when
the system is fine is as useless as one that cannot fail). Three days is long
enough that silence is unusual and short enough that a return hop broken on
Monday is visible by Thursday. It is the only signal that can show the last
hop failing: the puller can read the whole fleet perfectly, ask the ladder,
get a ruling, and hand it to nobody — and puller OK stays green through all
of it. That was the state of this rung before the decider shipped, and a
catalog row that could not show it would have called the gap healthy.
⚠️ THE SIGNAL IS THE puller OK LINE, NOT "A LINE WAS WRITTEN". The puller
writes puller PARTIAL when any source is UNREAD and puller DOWN when none
could be read, so a reader can see what is wrong — but neither matches the
pattern, so the row goes stale exactly when the puller stops being able to read
the fleet. Pairing it with digest fresh is deliberate: a job that dies and a
job that runs blind are different failures needing different fixes, and a
single signal that covered both would name neither. A signal matching any line
would be file: exists in another costume (ADR-0009).
⚠️ EXIT 0 MEANS THE PULLER RAN, NOT THAT THE BOARD IS FINE. launchd's
LastExitStatus says whether the job completes; the log word says what it
found. Non-zero happens only when the digest itself could not be written. A
puller that exited non-zero on a red PR would make "the puller crashed" and
"CI is red" the same colour, and those need different people.
Ladder routing — it routes, it never resolves. An open block with no ruling
goes to fable-decide — ⚠️ asked on BOTH model rungs concurrently, with
agreement driving and a disagreement settled by one Fable reconsideration
(ADR-0019).
This paragraph used to read "Fable → Astra → a human"; that ordering is
ADR-0016's and is superseded — it survives only inside a single call, as the
fall-through when Fable cannot be reached. Capped at 2 per tick and
ledgered by block id plus a hash of the question. A RESOLVED receipt is
reported in the digest as apply it with drive --task, addressed to the
Driver. ⚠️ Reporting a ruling in the digest and DELIVERING it to the
session parked on the question are two different acts — the delivery message
goes to the raiser and never to the Driver
(ADR-0020); this
sentence is about the digest only, and a reader who collapses the two will
think a ruling reached the blocked lane because the Driver saw it. A
question the ladder was reached on and declined is terminal and never
re-asked — ADR-0016's
rule, extended to the thing that feeds the ladder, because a scheduled job that
re-asks a refusal every five minutes launders it by attrition.
⚠️ THE ROW IS NOT GREEN UNTIL THE PLIST IS BOOTSTRAPPED, AND THAT IS THE
SIGNAL WORKING. One hand-run tick seeded ~/.claude/logs/puller.log and the
digest at 2026-09-10 18:39 local, so the Driver has something to read
tonight — but nothing is scheduled to write the next one, so puller OK ages
past its 900s window and the row goes stale, while launchd reads down for
a job that was never loaded. Both are true statements about a tool that is
built and not yet installed.
That is the property being bought, demonstrated on itself: the signal decays
when the thing stops running. A catalog row that stayed green on a system
nobody installed — or on one whose last tick was hours ago — is the exact lie
ADR-0009 exists to prevent. Bootstrap it (the command is in
~/.local/bin/README.md) and the row earns its colour on the next tick.
Appendix — everything linked from this entry
- docs/adr/0017-the-driver-is-fed-it-does-not-poll.md
- docs/adr/0019-two-rungs-are-asked-on-one-snapshot-and-agreement-drives.md
- docs/adr/0020-a-ruling-goes-back-to-whoever-is-blocked.md
- docs/adr/0030-every-pr-has-one-owner-per-job-and-it-is-armed-by-the-pr.md
- https://github.com/gera-propflow/local-bin#readme — the README
astra-relaunch
| What it does | Launches the next queued Astra brief when the lane goes to zero — a 300s launchd tick that derives liveness from the process table, claims one brief atomically, spawns a detached run under a per-tick, per-day and per-run cap, and exits; it never invents work and never loops |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/astra-relaunch (launchd co.propflow.astra-relaunch, 300s); queue ~/.claude/jobs/astra/queue/; receipts ~/.claude/jobs/astra/receipts/<runid>.json; run output /tmp/astra/<name>.log |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → astra-relaunch |
Liveness signals
| Signal | How it is read |
|---|---|
| launcher scheduled | launchd · co.propflow.astra-relaunch |
| the tick completed and read every source | log-match-age · ~/.claude/logs/astra-relaunch.log · matches astra-relaunch OK · stale after 20 min |
Detail document: ~/.claude/jobs/astra/run.json (the Systems tab opens this row into it).
This exists because the lane's supervisor was a Claude session's memory.
Until 2026-09-10 an Astra job ran only because some session hand-typed nohup codex exec -m gpt-6-astra …. There was no launchd job, no cron and no loop, so
a one-shot run exiting left the lane at zero until somebody remembered. It sat
at zero for 11.5 hours that day: the 15:00Z run finished normally, artifact
delivered, 215k tokens, and nothing launched the next brief. If a fact has to
be remembered to stay true, it will be false.
The astra-relaunch OK pattern, not any line. UNREAD and BREAKER lines
are written too, so a reader can see what is wrong — but they do not match, so
this row goes stale exactly when the rung stops being able to do its job. A
signal matching any line would be file: exists in a costume: green while
documenting failure. staleAfterS is 1200 — four ticks — so one slow tick is
not an outage and a stopped job is caught inside twenty minutes.
⚠️ THE LIVENESS SIGNAL IS THE LAUNCHER, NOT THE LANE, AND THAT IS DELIBERATE.
A green row here means the rung is deciding every five minutes, not an Astra
run is in progress. An empty queue is a correct, healthy IDLE — this rung
launches what is queued and never invents work, so keying the signal on "a run
is live" would render a legitimately quiet lane as an outage, and would go red
for the one reason nobody should be paged about. Whether the queue has anything
in it is the enqueueing session's problem; astra-relaunch --status prints it.
⚠️ detail CARRIES staleAfterS: 0 ON PURPOSE. run.json is an index into
the process table, not a fact — the rung reads a pid and a start time from it
and believes the lane is occupied only when ps agrees the pid exists, carries
that run's --wrap <runid> sentinel, and started when the record says. An
absent run.json means the lane is free; a stale one is reaped on the next
tick and logged as reaped. Nothing keys health off its age.
Appendix — everything linked from this entry
Agent Smith
| What it does | The fleet's workplace actor — Slack presence, PR reviews, ticket intake |
|---|---|
| Kind | actor |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | — |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Agent Smith |
Liveness signals
| Signal | How it is read |
|---|---|
| worker loaded | launchd · co.propflow.smith-worker |
| slack socket loaded | launchd · co.propflow.smith-slack-socket |
An actor, not infrastructure: it uses the fleet's rails (credentials from the pipeline above, sessions like any other) rather than providing rails.
The worker's own starvation alarm (agent-smith #736, 2026-10-03). The
Temporal worker watches itself for the condition that precedes a wedge: its
event loop too starved to relay heartbeats. It counts the SDK's
[TMPRL1104] Workflow task exceeded 10 seconds warnings plus
agent_smith.heartbeat's "heartbeat not accepted by the worker loop" lines;
8 in 300 s posts ONE #alerts message per episode, titled
Smith worker STARVED — host load N/core, and writes
watchdog: N loop-starvation lines in 300s … at ERROR to
~/Library/Logs/smith-worker.err.log. It never exits (the separate wedge
watchdog still does, on failed cache evictions), re-posts at most hourly, and
is not gated on uptime. Why it exists: on 2026-10-02 (21:26 and 22:41 CDT) the
host ran the box to swap 21 GB / load ~100 on 14 cores, three brain turns
timed out with their claude -p children still working, and nothing told a
person until the wedge exit ten minutes in. The alert is from INSIDE the
worker — loop lag and rejected heartbeats — and complements the host-side
host / fleet-stability / pacer rows, which measure the box; it does not
measure the box itself beyond quoting os.getloadavg() in the title.
⚠️ Deliberately NO liveness signal for it here. smith-worker.err.log's
readable tail is ~11 minutes (see the ci-runs-on-canary note below), so a
log-match-age on the boot-time watchdog: armed line would go stale on
every healthy long-lived worker, and one on the alert line itself would read
stale = healthy — the inverse of what this table means by stale. The alarm's
proof of life is the worker loaded signal above (the detector is installed
in worker.run, not a separate process); its proof of function is the
#alerts post. If a real signal is wanted, the detector needs its own
periodic line in its own log first.
Appendix — everything linked from this entry
Smith owed-ask ledger + 5-minute chase
| What it does | Every human ask Smith receives is written down at the door (agent_harness.inbound_asks, ~/.claude/smith-state/inbound-asks.db) and stays OWED until a real answer (or a deliberate decline) closes it; a failed or empty run does not. Every 5 minutes the Slack socket chases the owed ones (agent_harness.owed_asks): owed >15 min with no turn running on the thread → re-run the turn once through the same dispatch door; still owed 15 min later → one top-level #alerts page tagging Gera (and Fede when he is the one waiting), linking the message. The morning Open-loops digest and /smith-review (`python -m agent_harness.inbound_asks owed --days 14`) read the same ledger. Since ADR-0080 the same ledger file and the same 5-minute tick also hold OWED ITEMS beyond replies (agent_harness.owed_items): an expired PR drive, a prompt escalation with no decision row, a nightly check red 3+ nights with no owner, an expired task or decision, a claimed bridge handoff. Each stays open until its kind's proof (PR merged or closed, row answered, check green or muted or retired, a later DONE, a person's close) and pages once when it passes its bound: in Gera's DM with no tag when the stall is Gera's alone, or in #alerts tagging Gera (and Fede when it is his, a customer handoff included) otherwise, and always in #alerts from a seat other than Smith (ADR-0080 amendment 2026-11-04) |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | Ledger since 2026-09-06; only-an-answer-closes is agent-harness #1791, the honest failure notice #1792, the 5-minute chase #1793 (runs inside co.propflow.smith-slack-socket, armed by its restart — no Schedule to register or unpause); owed items beyond replies are ADR-0080, recorded at the closure transport (activities/slack_send), automation_run and the bridge handoff intake, swept in the same tick |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → Smith owed-ask ledger + 5-minute chase |
Liveness signals
| Signal | How it is read |
|---|---|
| owed-ask chase sweeping | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches owed_asks.sweep read= · stale after 30 min |
| owed-item sweep reading | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches owed_items.sweep read= · stale after 30 min |
| slack socket loaded | launchd · co.propflow.smith-slack-socket |
Why it exists (2026-11-01). Fede asked Smith twice in #alerts at 16:21Z.
Both runs were refused by the session lease, both asks were written
empty_run, and empty_run used to count as an ending, so the unanswered
sweep skipped them. That sweep also ran only once a day, at 07:00, into a
digest. Nothing paged for 8+ hours. In the 14 days before, six asks closed
as empty_run and none was reported. The failure notice was false too: it
said "its process exited on its own" when the lease had refused the run.
The signal. owed_asks.sweep read=… is logged only at the END of a sweep
that read the ledger. A sweep that throws logs owed_asks.sweep_failed
instead, which this pattern does not match. 1800 s is six missed ticks. The
lines open with the logging stamp (2026-11-01 19:59:05,892 INFO …), which is
the form log-match-age reads. ⚠️ The entry reads stale until the socket is
restarted onto #1793, which is why it is proposed. Flip it to active once
the line is seen live.
Owed items beyond replies (ADR-0080, 2026-11-01). Gera: "handled" needs
proof and a stall pages. The same rule now covers five more kinds, in one
owed_items table in the same file. Each has its own proof and its own bound:
an expired PR drive (PR merged or closed; pages 6 h after the EXPIRED line,
naming the one blocker GitHub reports), a prompt escalation with no decision
row (a row for that session answered; 2 h), a nightly check red 3 nights with no
open fix PR (green, muted by a person, or retired; at once), an expired task,
operator task or decision (a later DONE, or a person's close; 24 h), and a
claimed bridge handoff (done, or released to a named taker who claimed it;
72 h). A FAILED or EXPIRED line never closes an item. Each item pages #alerts
once, claimed in its row before the post. python -m agent_harness.owed_items owed lists the open ones; … close KEY --by NAME --proof TEXT is a person's
close. The second signal reads the sweep's end-of-pass line, which is logged
only after the owed-item sweep slice is live on the socket.
Not a second open-loops system. The morning loop_closer Open-loops
section, the ack watchdog (dangling_acks) and this chase all read Smith's
own stores. This entry is the INBOUND half: it starts from what a person
asked, which is the only place a silent drop is visible.
Appendix — everything linked from this entry
prod-write (harness-prod-write)
| What it does | The one door a production change goes through: a named operation (an SDK call or a literal argv), fingerprinted over exactly what is called. It always needs Smith's written certainty check (the exact change, a dry-run or read-back plan, how it verifies afterwards), plus EITHER Gera's own go on that fingerprint and decision (read from the decision answer store, Gera's address only) OR Trinity's independent yes (a parsed bridge message from her bot user). It records a receipt that spends the approval, whichever path gave it |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | harness-prod-write console script (agent_harness.prod_write_cli, agent-harness #1646) and its executor (ProdWriteWorkflow in src/agent_harness/workflows/prod_write.py, agent-harness #1654) on Smith's worker; path B (two seats) since #1711, path A (Gera's go, read through `blocked answer`) since #1716, live on the worker from 2026-10-29T02:31Z (boot sha b134ae4e); receipts (the spent-approval record) in ~/.claude/smith-state/prod-write/receipts.jsonl, which the child can delete (ADR-0075 amendment, item 6) |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → prod-write (harness-prod-write) |
Liveness signals
| Signal | How it is read |
|---|---|
| worker boot passed the tool's self-check | file-age · ~/.claude/smith-state/prod-write/selftest.stamp · stale after 4320 min |
Check here before giving any agent a production write. Since ADR-0075 (amended 2026-10-28 and 2026-10-29) a production change is not a command an agent runs. It covers an AWS write, a GitHub Actions variable or secret, a merge-queue jump, and a Vercel production deploy, promote, alias or env change.
How a change gets approved. It is a named harness-prod-write operation. Smith's certainty check is always required: the exact change and its fingerprint, how it was dry-run or will be read back, and how it will be verified afterwards. Then EITHER:
- Gera's go: his own answer of
Go: run change <fingerprint>on that decision, on the docs decision page (the answer store records who answered). Only Gera's address counts: not Fede's, not Sean's, not an agent's, not a resolved ledger row; - or Trinity's yes: an independent approval of that exact fingerprint and decision, received as a parsed bridge-protocol message from her bot user in
#engineering-bridge.
Free text never counts ("Gera said go", "Trinity approves"). Gera's rule (2026-10-29): "either my go or if Trinity and Smith both agree". Withdrawing the decision stops a change on either path. Gera's go counts when clicked on the docs decision page or the /agents Decisions tab. An /agents click is recorded under the propflowai server's Access service token with a ::gera@propflowai.co key, and Gera ruled on 2026-10-29 that this counts, accepting that anything holding that token could approve in his name (agent-harness #1731, merged as b84d5565; live once Smith's worker boots on it, per its boot record). Until Trinity's seat is updated (Fede), Gera's go is the only path that can run a change.
What this is not. The credential cutover of the original ADR is dropped: the write credential stays where Smith's child can reach it. So this row enforces the approval for changes made through the tool. It is not a wall against a child that runs a write directly.
The signal reads a stamp Smith's worker writes at each boot after the tool's self-check passes (harness-prod-write selftest, agent-harness #1663, merged and live, so the row is not stale for lack of a self-check).
- What the self-check does: it fingerprints a synthetic change, finds every operation's SDK method and CLI program, and checks the receipts directory. It runs nothing and reads no network.
- When the stamp is written: a boot happens at every deploy, so stale means three days with no worker boot that could find the tool.
- Why the receipts file is not the signal: it is written only when a change runs, so it would be stale on a healthy quiet week.
- Status: the row stays
proposeduntil a change has run through the tool on either path.
Appendix — everything linked from this entry
Agent registry (config/agents.toml)
| What it does | The one list of agent seats: every seat loads every row, so each agent knows its peers; this machine's IOPlatformUUID names its row, and both daemons log at boot whether the process agrees with it (WARN mode until enforced) |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-smith |
| Deployed at | config/agents.toml + agent_smith/agent_registry.py, read at import by config and bridge_claims; boot check in worker.run and slack_socket.main; row in smith doctor |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → Agent registry (config/agents.toml) |
Liveness signals
| Signal | How it is read |
|---|---|
| worker boot agreed with this machine's row | log-match-age · ~/Library/Logs/smith-worker.err.log · matches registry boot-check OK · stale after 4320 min |
| socket boot agreed with this machine's row | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches registry boot-check OK · stale after 4320 min |
Before you add an agent, add its row. A new seat is a reviewed PR to
config/agents.toml (plus its Slack app's bot uid and its
config/mcp-allowlist.<slug>.json). Nothing else lists seats. A boot that logs
registry boot-check MISMATCH (warn only) means the process and its machine's
row disagree. The signals read PRESENCE of the OK line, since a boot only
happens on a deploy or a restart, and stale means no daemon on the RELAY'S
host (the mini) has agreed with the registry for three days. Another
machine's seat (Trinity's) logs on that machine and is not seen here. It flips
to active once registry part 2 (agent-smith #1365) is live. ADR-0072.
Appendix — everything linked from this entry
Seat heartbeats + sibling check-ins
| What it does | Every agent seat writes one shared DynamoDB row (boot commit, boot time, last-turn time, host, protocol number) at boot and every 5 minutes; each seat reads its sibling's row before a handoff or cross-review and every ~15 minutes, and a stale or behind sibling gets one bridge reminder, then its machine's person, then the checker's person, once per episode |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_harness.seat_heartbeat, a daemon thread started by worker.run after record_boot, in co.propflow.smith-worker (and co.propflow.clara-worker on Fede's machine); row PK=SEAT_HEARTBEAT#<registry slug>, SK=#LATEST in propflow-prod; section in smith doctor |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Seat heartbeats + sibling check-ins |
Liveness signals
| Signal | How it is read |
|---|---|
| this seat publishing | log-match-age · ~/Library/Logs/smith-worker.err.log · matches SEAT-HEARTBEAT published · stale after 20 min |
Why it exists. On 2026-11-01 three cross-seat protocol changes landed in
one day: the reviewer's risk label, owed asks and restart safety. Trinity
follows each one only after Fede pulls and restarts her, and the mini had no
record of her running commit. Each worker's boot record stayed in its own
state dir, which is why fleet-deploy reads her UNKNOWN. The shared row is the
transport ADR-0011 deferred. A seat emits it and the other seat reads it, and
nothing calls the other seat (ADR-0003).
ADR-0082.
Appendix — everything linked from this entry
Alert IDs + derived fix history
| What it does | Prints a stable Alert ID (source, or source/class) on every marked alert post, and before Smith's fix engines attempt a fix, derives that ID's past fix PRs, recent verdicts and recurrences, so a second attempt must explain why the first did not hold and a third becomes a design doc |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.alert_history, called from slack_socket._route_alert, in co.propflow.smith-slack-socket; footer in propflowai notify.ts (alert-id.ts) and agent-smith notify.py; cache ~/.claude/smith-state/alert-history-cache.json |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Alert IDs + derived fix history |
Liveness signals
| Signal | How it is read |
|---|---|
| socket loaded | launchd · co.propflow.smith-slack-socket |
| history lookups happening | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches alert_history lookup id= · stale after 1440 min |
Why it exists. Seven merged Smith fixes for identity_contact_match_refused
in two weeks, the class re-firing after each, and no attempt asking whether the
last one held; across 10-05..19, 43 of 69 alert-fix PRs were bandages and 28 of
those alerts fired again (ADR-0070).
The ID is the key that already existed (source), now printed. The history is
read from GitHub and the outbound ledger, not stored.
Appendix — everything linked from this entry
Smith/Trinity team protocol
| What it does | Gives every Smith/Trinity thread one driver seat, shown as a sticker reaction on the thread's top message (picked by area: Gera's work -> Smith, Fede's -> Trinity, Smith when unsure, a hand re-sticker overrides), a driver card on driven work, and turn-taking: each seat stamps working/final on its own messages and never answers over the other seat's unfinished stub |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.slack (turn marker, chat.update metadata), activities/slack_send + workflows/reply (stamping, in co.propflow.smith-worker), slack_socket._dispatch_trigger and the thread-participant block (the wait and the router, in co.propflow.smith-slack-socket). Live on smith (the mini) since 2026-10-20; clara (Trinity) runs it once Fede's machine pulls the same main (pending at acceptance) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Smith/Trinity team protocol |
Liveness signals
| Signal | How it is read |
|---|---|
| socket loaded | launchd · co.propflow.smith-slack-socket |
| worker loaded | launchd · co.propflow.smith-worker |
| threads getting a driver | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches thread_driver chan= · stale after 2880 min |
Why it exists. 184 of 554 findings in the 2026-10-19 Smith strategy review (33%, the largest root cause) came from no single owner per thread: both seats answering one alert (09-24: 8 of 12 Sentry roots, 6 conflicting), then, after the 10-12 "first seat to post owns the thread" fix, neither answering (16 of 66 human messages unacked 10-16..19). And each seat replied to the other's stub because a stub and an answer looked the same (ADR-0071).
The reviewer labels the risk (Gera 2026-10-31,
ADR-0078).
A CROSS-REVIEW REQUEST asks the other seat for a verdict and a
Risk: low|med|high, and puts hold-for-review on the PR. The reply is
recorded in code (cross_review.record_verdicts), and the PR drive's gate
decides: ROOT with low or med lifts the hold and merges on green + 🟢; anything
else, or no label in time, pages both founders. A person's own hold is not
lifted (one stated exception: a label the PR's author adds within 30 s of
opening reads as the driver's; ADR-0078 §3).
Asking every agent at once: @Agent Force <question>, the one door
(Gera 2026-10-30; agent-harness #1752, #1753, #1757,
ADR-0077).
A person's message that @-mentions the Agent Force bot user, top-level or in a
thread, is an ask of every seat in that channel (agent-harness
agents_ask.mention_ask → slack_socket._answer_agents_ask); the question is
the text with the mention stripped. Each seat finds Agent Force's uid by name
(users.list, users:read; AGENT_FORCE_USER_ID overrides). Nothing is posted
for the ask itself: in a thread the answer lands in that thread with its
earlier messages as context.
Both seats work it. The pick, in order: a seat the question @-mentions alone;
in a thread, the thread's driver (the follow-up chain, so a hand sticker on the
root wins); else the area rule on the asker (thread_driver.ask_pick, one
function for both seats and the dispatcher below). The driver answers and puts
its sticker on the thread; the other seat waits the owner window, then waits
for the driver's answer to turn final and adds a correction, a missed risk or a
missing step in the thread, or nothing. The driver re-reads a late ask so it
never answers twice. Only the agents of that channel take part: a pick naming a
seat that is not a member is replaced by the seat holding the ask, at once
(slack.channel_member_ids, a failed read keeps the window). Once the
dispatcher posts its pick, the seats take their roles from that post and fall
back to their own pick when none arrives. Agent Force is never a seat and never
answers. Liveness is the threads getting a driver signal above; an ask logs
thread_driver chan=… source=… (agents ask=…). Not in DMs or #homepage, where
the ordinary path already answers.
Retired the same day: /agents and agents: (Gera, 2026-10-30: "That was
just like a hack. I really like the Agent Force idea a lot better."). The
/agents slash command (2026-10-28, #1718/#1719/#1722) and the agents:
message prefix (its in-thread stand-in, #1744) are gone: two doors to one
engine would be a switch. A /agents Slack still delivers before the command is
deleted in Smith's app config gets one ephemeral line pointing at @Agent Force.
Old /agents thread roots still read as their asker's, so those threads keep
their driver.
Appendix — everything linked from this entry
Agent Force dispatcher
| What it does | Receives an @Agent Force mention on Agent Force's own Slack app (Socket Mode, app_mention), makes the ONE pick of which seat drives it (thread_driver.ask_pick, then the channel's membership) and hands it to the seats as one marked thread reply, '<@driver> drives · <@other> adds after'. The seats obey the post and fall back to their own pick when it does not come. Agent Force never answers a question |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_harness.agent_force_dispatch, launchd co.propflow.agent-force-dispatch on the mini (deploy/co.propflow.agent-force-dispatch.plist, scripts/install-agent-force-dispatch-launchagent.sh). Tokens AGENT_FORCE_BOT_TOKEN / AGENT_FORCE_APP_TOKEN in propflowai/.env.local, read via agent_harness.config; either unset → one log line and the process idles. Part 2a logs its pick and posts nothing; part 2b posts it |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → Agent Force dispatcher |
Liveness signals
| Signal | How it is read |
|---|---|
| dispatcher loaded | launchd · co.propflow.agent-force-dispatch |
| picks being made | log-match-age · ~/Library/Logs/agent-force-dispatch.err.log · matches agent_force_dispatch pick chan= · stale after 10080 min |
Why it exists. Gera, 2026-10-30: Agent Force is a dispatcher he can @ in Slack's autocomplete, that visibly decides who drives and gets both agents working together (ADR-0077). Without it each seat makes the same pick on its own, which holds only while both run the same code.
Appendix — everything linked from this entry
gera-tabs (/gera-tabs → the numbered tab table)
| What it does | On demand in Slack, posts Gera's open Claude tabs as one numbered table — #, label, sid8, attached/dropped, last active, machine (mini/mac) — derived from the MINI's own ledger (~/.claude/mini-tab-ledger.json) so it answers with the Mac asleep, then optionally enriched over `ssh mac` from `open-top.py --tabs --json` for the true left-to-right iTerm order and any session running on the Mac itself. The footer always states WHICH of the two orders is in use. Numbered FROZEN per (channel, thread_root) so `#1` means the same session on the next turn. Nothing is opened and nothing is sent. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.gera_tabs in the main checkout ~/code/PropFlow/agent-smith. TWO doors, ONE handler (`post_table`): the plain-text `gera-tabs` / `tabs` / `@Agent Smith tabs` reaches the brain, which runs `smith-gera-tabs --post`; and since agent-smith#960 the SLASH command `/gera-tabs` is ACKed by `slack_socket.handle_envelope` and routed by `slack_socket._handle_slash`, which gates on the command name (`_GERA_TABS_COMMANDS`) and on Gera's uid, then hands `gera_tabs.post_table` to a daemon thread so the socket pump is never held by an `ssh mac`. Reads ~/.claude/mini-tab-ledger.json (the source — written by co.propflow.mini-tab-ledger, which is a system of its own). State: ~/.claude/smith-state/gera-tabs-numbering.json (the frozen per-thread order) and gera-tabs-feed.json (the Mac cache, now reached only on a box with no ledger). |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → gera-tabs (/gera-tabs → the numbered tab table) |
Liveness signals
| Signal | How it is read |
|---|---|
| surface hosted | launchd · co.propflow.smith-slack-socket |
| the Mac's feed has answered | file-age · ~/.claude/smith-state/gera-tabs-feed.json · stale after 20160 min |
⚠️ THE SOURCE CHANGED ON 2026-10-13 (agent-smith#961), AND TWO CLAIMS ABOVE
WERE CORRECTED IN THE SAME EDIT. Gera: "the tool LIVES ON THE MINI and must
NOT depend on the Mac. The Mac will usually be asleep." The table is now
derived from ~/.claude/mini-tab-ledger.json and the Mac is an enrichment on
a 3-second connect leash (8s overall, down from 8/30 — they used to bound the
only source, now they bound a nice-to-have, and the live round trip measures
2.6–2.7s). The two corrections: this row said the /gera-tabs SLASH command
"is routed by slack_socket._handle_slash" — there was no such function when
that was written; handle_envelope ACKed slash_commands and ignored them, and
#959's own docstring says the slash door was cut out of that change. And the row
named gera-tabs-feed.json as the fallback the table rests on; it no longer is —
see below.
⚠️ THE FIRST OF THOSE TWO CORRECTIONS IS ITSELF STALE, AND STOOD FOR ~4.6
HOURS. slack_socket._handle_slash EXISTS — agent-smith#960 landed it at
2026-10-13T06:42:14Z, which is 01:42:14 CDT (gh pr view 960 --json mergedAt, and the receiver's matching restart is at 01:42:31 in its log). The
sentence above was written against #959 before that and was not revisited; it
was still standing when this was measured at 06:17 CDT, 4.6 hours later. (It
was first written up as "~9 hours" — that is 01:42 read as UTC against a CDT
clock. Every time in this entry is now suffixed, because the field two above
quotes 03:38Z and an unlabelled hour between them is a trap.) It is kept rather than deleted because it is true
about #959 and dating it is the only way the next reader can tell which era a
claim belongs to; but read as a present-tense fact it said the slash door was
unimplemented, which is worse than saying nothing: this catalog is the
fleet's map, so a row that positively asserts a function does not exist tells
you to stop looking — exactly the wrong instruction while the open question is
why /gera-tabs has never arrived. See deployedAt above for what the function
actually does.
THE TABLE IS THE UNION OF TWO SOURCES THAT SEE DIFFERENT SETS. Measured
against both, live, 2026-10-13: 8 sessions in both; 2 in the ledger only
(op-gera-tabs-slack-relay, op-spending-brief-fix — real Mac tabs, both
mac_clients: 1, but in a different iTerm window from the one --tabs
derives); 1 in the feed only (ccswitch, machine: "mac", running on the
Mac itself with no mini tmux). Neither source alone is the answer: the ledger
alone makes the machine column say mini for every row, and the feed alone
drops two sessions Gera has tabs onto. A merged row keeps the ledger's
identity (sid/tmux/label, joined on tmux) so the frozen #N does not
move when the Mac goes to sleep; only state and the ordering come off the
feed.
gera-tabs-feed.json's signal now proves LESS than it did, which is why it
is kept and relabelled rather than removed. It is written only by a live Mac
read that succeeded, so it still proves the ssh path works — but the table no
longer rests on it. It is reached on exactly one branch: a box where the ledger
FILE IS ABSENT (no recorder installed), which is not the same as a recorder
that died. A broken or stale ledger deliberately does not fall back to the
Mac, because that would hide a dead recorder on every turn the Mac happened to
be awake. 14 days, not hours, because the surface is DEMAND-DRIVEN: nobody
schedules it, so a week where Gera did not ask is quiet, not sick.
The call is guarded before it runs, and the guard is not vestigial. An older
open-top.py ignores a flag it does not know and ACTS: measured 2026-10-13
03:36Z, open-top.py --help on that Mac did not print help, it opened four
real iTerm tabs. The hazard is closed on the Mac now, but the two machines
are not the same: the MINI's copy does NOT speak --tabs while the Mac's does
— and the Smith worker runs on the mini. (Measured 2026-10-13, and the
numbers are the symptom rather than the rule: 1076 lines against 1225. The rule
is that the mini's copy lags, so read the flag, never the line count.)
~/.claude lags
origin whenever its updater stops on a local edit, so the installed copy is
never the copy you read, on either box. The guard and the invocation therefore
sit inside ONE remote command: grep the Mac's file, and exec that same file, or
exit 3. A guard that checks the local copy and then runs the remote one proves
nothing about what it ran. A missing flag means fall back to cache, never
run it and see.
Silence is never rendered as an answer. ssh hands back the remote exit code
and keeps 255 for its own failures, so one round trip separates 0 JSON from 3
feed-too-old, 4 no-feed, 255 unreachable and anything else. Only the first
says how many tabs exist. The other four serve the last good listing stamped
"as of <time>", and with no cache at all the message says it could not
reach the Mac and says in as many words that this is not it saying there are no
tabs. "has no tabs in it" is reserved for a read that succeeded and returned
zero rows.
Reach was verified in the launchd environment, not just a shell. ssh mac
succeeds from a byte-for-byte replay of the live slack-socket daemon's environ
(read off ps eww), from the worker plist's environ, and from a variant with no
SSH_AUTH_SOCK — so the key is passphrase-less and there is no ssh-agent
dependency. Negative control (env -i HOME=/nonexistent ssh -F /dev/null) is
refused, so the positive is not vacuous. It is a replayed environment, not an
exec from inside the running daemon.
⚠️ ONE HUMAN STEP, ONCE, for the slash door. api.slack.com → the Agent Smith
app → Slash Commands → Create New Command → /gera-tabs, then reinstall the app
when Slack prompts. There is no Request URL: the app is Socket Mode, so the
command arrives on the WebSocket that is already open and nothing is deployed
publicly. Until that click the plain-text doors work and /gera-tabs simply
never reaches us.
AND THAT CLICK IS STILL THE OPEN QUESTION — measured on this box
2026-10-13 06:17 CDT. ~/Library/Logs/smith-slack-socket.err.log is
22,864,760 bytes reaching back to its first line at 2026-06-30 14:27:37, and
grep -c slash_commands over the whole of it returns 0. The receiver is
alive and taking traffic (last restart 04:56:43, 45 lines since, 41 of them
rx type=file_change and nothing else; mtime 7 minutes old at the time of the
read). So no slash command has ever reached this app, and the human step above
is the first thing to check.
WHAT THE LOG COULD NOT TELL YOU, AND WILL — agent-smith#967, board row
slash-door-is-diagnosable. ⏳ RETIRE ONCE /gera-tabs HAS ANSWERED ONCE:
everything from here to the end of this section is true as of 2026-10-13
and describes a door that has never been used. The moment the log shows a
slack_socket slash /gera-tabs line, the four-outcome table below has done its
job and this whole block should collapse to one sentence saying the door works
and when it first did. A diagnostic kept past its diagnosis reads as a standing
problem. The unrecognised-command drop in _handle_slash
logged at DEBUG, and DEBUG is off in the deployed daemon, so a command that DID
arrive and that this receiver does not own was ACKed to Slack and left no trace
at all. #967 raises that line to INFO carrying the command name and channel. It
is in the docs list above; it changes nothing until the receiver next
restarts on the checkout it is deployed from (the restarts at 01:42:31 and
04:56:43 CDT on 2026-10-13 are in this log, so this happens unattended — but a
merge is not a deploy, and this log is the only way to tell which code is
running).
⚠️ AND THE LOG IS ONLY HALF THE EVIDENCE — the other half is in Gera's DM. Socket Mode delivers only the commands registered on this app, so the four outcomes of his one attempt are distinguishable only by reading both.
EVERY ROW IS LABELLED BY WHAT BACKS IT, because this is the row where that
distinction gets lost. ⓞ = observed or pinned by a test in
tests/test_slack_socket.py; ⓘ = inferred from Slack platform behaviour and
NOT watched in this workspace. The inferred rows cannot be watched from here:
Smith must not post into that DM, so the only person who can see Slack's reply
is Gera. The table is therefore also the experiment — have him copy down the
exact text Slack shows him rather than trusting these labels, and fold the
answer back into this entry.
| he sees in the DM | the log shows | what it means |
|---|---|---|
| the table | slack_socket slash /gera-tabs chan=… user=… |
ⓞ it works |
| nothing | that line, then /gera-tabs failed (already acked) |
ⓞ the post failed AFTER the ack — e.g. not_in_channel |
| nothing | /gera-tabs from U… — not the owner |
ⓞ the uid is not OWNER_SLACK_USER_ID |
| nothing | ack+ignore slash_commands /<name> chan=… |
ⓞ registered under a name _GERA_TABS_COMMANDS does not have — fix the registration (the line agent-smith#967 makes visible) |
| Slack's own "not a valid command" | nothing | ⓘ the name he typed is not registered on this app — do the click above |
dispatch_failed or operation_timeout |
nothing | ⓘ registered, but no receiver ACKed inside Slack's ~3s window (no Socket Mode connection open, or one open and nothing acked) |
| nothing | nothing | ⓘ never sent — or another Socket Mode connection on the same app token took the envelope. Slack spreads envelopes across every open connection, so a stale or dev receiver elsewhere can swallow it silently |
⚠️ THE LAST ROW IS THE TRAP, and it is why "nothing happened" is never a diagnosis on its own. A second receiver on the same app token — a dev box, a forgotten tmux — would take some envelopes and ack them, and this log would stay empty while the command demonstrably worked. An earlier draft of this table said "nothing / nothing" meant a delivery fault, full stop; that was wrong in a way that would have sent someone to check the socket while the real answer was a second listener. (Corrected by agent-smith#967's round-3 review, which also corrected its OWN round-2 table — the review could not fetch Slack's docs from its runner, hence ⓘ rather than ⓞ on the rows that rest on them.)
That is the whole point of the row: Smith must never post into Gera's DM (ADR-0051), so his single attempt is the only experiment available and it has to leave evidence either way. The two-column reading is agent-smith#967's round-2 review correcting this row's first draft, which said a wrong-name registration produced "the same silence as one never registered" — it does not: that case never reaches us and Slack answers him directly.
Appendix — everything linked from this entry
- https://github.com/PropFlow-Technologies/agent-smith/pull/959
- https://github.com/PropFlow-Technologies/agent-smith/pull/960
- https://github.com/PropFlow-Technologies/agent-smith/pull/961
- https://github.com/PropFlow-Technologies/agent-smith/pull/967
- https://github.com/gera-propflow/dotclaude/pull/244
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README
gera-tab-relay (tell #N / summary on N)
| What it does | Turns the numbered tab table into two verbs: `tell #N <text>` forwards Gera's words into that session through the sanctioned `msg` CLI with `--force` (mid-turn, by his own choice) and confirms in ONE line what went where; `summary on N` recaps what that session has been doing from the tail of its OWN transcript JSONL. Both resolve `#N` by CALLING `gera_tabs.order_for`, so a number means the same session here as it does in the table, and both refuse rather than guess. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | MERGED AND LIVE ON BOTH SURFACES, measured 2026-10-13 on the MINI (MiniClaws-Mac-mini, UTC-0500; every time below is UTC). On main: `tell #N` is agent_smith.gera_relay (`smith-gera-tell`), agent-smith#965 merged adc7e523; its owner gate is agent_smith.hooks.command_gate, #966 merged 62fe8d3e, which inverted to deny-on-mention after four generations of enumeration; the misroute path it depended on is closed by agent-tools/resume-any, #971 merged ff801b5f; and `summary on N` is agent_smith.gera_recap (`smith-gera-summary`), #976 merged ded529ade. Reads ~/.claude/projects/<cwd-slug>/<sid>.jsonl for the recap, bounded to the last 2 MB because the largest transcript on the MINI is 932 MB; writes nothing of its own. ⚠️ 'DEPLOYED' MEANS TWO DIFFERENT THINGS FOR THIS SYSTEM, AND A MERGE DATE ALONE PROVES NEITHER - that is the claim that had to be retracted to three lanes on 2026-10-13. (1) THE CLI SURFACE NEEDS NO RESTART: a console script imports from the checkout AT INVOCATION, so merged code runs the next time the brain calls it. Receipt: .venv/bin/smith-gera-tell and .venv/bin/smith-gera-summary both present and executable, src/agent_smith/gera_recap.py mtime 2026-10-13T16:57:06Z - seven seconds after #976 merged at 16:56:59Z, so something syncs the checkout at merge time - and checkout HEAD cde4451c contains all four merge shas. (2) THE IN-PROCESS SURFACE NEEDS A RESTART, AND NOTHING RESTARTS IT ON A MERGE: neither co.propflow.smith-worker.plist nor co.propflow.smith-slack-socket.plist has WatchPaths (both are KeepAlive, i.e. restart on EXIT, not on code change), and there is no deploy script - so the in-process surface goes current incidentally, on the next crash or manual restart. Receipt: worker pid 96060 started 2026-10-13T17:12:09Z, AFTER all four merges (latest 16:56:59Z), so in-process code is current too; it was the third worker restart that day. TO RE-RUN THIS RATHER THAN TRUST IT: compare the worker's start time against the merge time ON ONE CLOCK - `ps -o lstart=,etime= -p <pid>` gives a LOCAL lstart that must be converted to UTC (-0500 here), while `etime` is elapsed and needs no conversion, which makes it the safer of the two; `gh pr view <n> --json mergedAt` is already UTC. Comparing a local lstart against a UTC mergedAt is exactly how this was got wrong once. For the CLI surface there is no process to time: read the source mtime and confirm the checkout HEAD contains the merge sha. |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → gera-tab-relay (tell #N / summary on N) |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
ADR-0002 picks the channel, and the rung matters. ADR-0002
retires typing into another session's pane for agent-to-agent traffic and names
the fallback order: SendMessage by name, then the msg CLI for text, then
wake-session for a slash command the peer must RUN. SendMessage is a Claude
Code tool and this is a Temporal worker with no turn to call it from, so the
rung actually reachable is the CLI - which rule 1 names in the same breath.
Addressed by full sid, because the ADR says so explicitly ("by name the
daemon picks a same-name fork first"); a row whose sid is a legacy 8-char
value is refused rather than guessed at. And a body starting with / is
refused: that is the wake-session rung, and the sender's own stamping
deliberately leaves a slash-command body unstamped so that it runs, which
makes it the one shape that would reach a session as an anonymous executable
command.
It refuses rather than resurrects, and that is the whole safety story. One
live copy of a session is the rule, so a forward that silently started one back
up is the worst available outcome - worse than no delivery. There is no second
liveness mechanism beside the sender: it already refuses a sid with no live
pane, exits non-zero, and never spawns. Measured against a sid whose tmux is
genuinely gone: exit 1, the tmux session count unchanged at 40, has-session
still failing, and no inbox directory created - so nothing was parked
either.
--force is what keeps the receipt parsing honest, which is not obvious.
~/.local/bin/msg is a symlink to a bash shim, not to the python tool in
agent-smith: a plain send is intercepted and routed over the daemon control
socket, which queues for a mid-turn target where the real tool parks.
Different program, different wording, different failure modes. The shim
re-execs the real tool with the original argv for any unrecognised --flag,
and --force is one - so the invocation provably lands on the file the drift
guards hold their literals against. If a later part ever drops --force, its
receipt parsing and its drift guards both silently stop describing the code
that runs.
The sender exits 0 for a delivery AND for a park, so the exit code cannot produce the sentence. Parking is a success - the text cannot be lost - but it is not what Gera asked for, because a parked line waits until that session next drains its inbox. The marker on the sender's first output line is the only discriminator it offers, so those two characters are drift-guarded against its source. There are four exit codes and the gap between two of them is the point: 0 reached it, 1 nothing was sent (safe to resend), 2 bad invocation, 3 it may have arrived or arrived somewhere else - never resend. A misroute and a timeout both land on 3, because by the time the sender prints its receipt it has already typed the text, and folding either into "nothing sent" is how the same forced line gets typed twice into a live session.
The provenance stamp is anonymous from a worker, so the relay names itself.
The sender prefixes [msg from <name> . <sid8>] using ra.own_sid(), which
resolves from $CLAUDE_CODE_SESSION_ID, then a tmux pane's
@claude_session_id, then process ancestry. A launchd-spawned worker has none
of the three, so the stamp degrades to a bare [msg] - enough to mark the
line as relayed mail, nothing about who relayed it, and identity-less
delivery is exactly what ADR-0002 retired the pane path over. The body
therefore carries [Gera's tab relay . #N], naming the mechanism and not the
person: "from Gera" would rest on an owner gate that an inline env prefix
satisfies, so a forged forward would arrive wearing his name.
The recap reads the transcript, and the ledger cannot hand over the path.
That was the part of this row that had to be solved rather than looked up: the
feed's resume_path is null for every connected tab by design - its own
docstring says "Null means NOT LOOKED UP, never 'no transcript'" - and
connected is the only state summary on N is ever asked about. So the path is
resolved from the sid instead, at
~/.claude/projects/<cwd-slug>/<sid>.jsonl. One root is enough, which is a
measurement and not an assumption: ccswitch gives every session its own
CLAUDE_CONFIG_DIR, but 337 of the 340 session-cfg/*/projects entries are
symlinks to that one root (2026-10-13). The read is bounded - the largest
transcript on this box is 932 MB - and the digest's header names the window
against the file's size, so a tail cannot be read as a history.
No /recap exists to reuse, and the meeting pipeline is deliberately not
bent to this. Searched first: commands/sum.md summarises the conversation it
is typed in; skills/explain re-frames work already in context and reads no
transcript; skills/search-convo does read these JSONLs (and is where the
path convention is confirmed) but searches by keyword across sessions and
drives the Grep and AskUserQuestion tools, so it is a turn's recipe rather
than a mechanism - it also documents session-memory/summary.md files that
would have been the ideal reuse, of which zero exist on this box. What is
reused is the JSONL-reading precedent only (bridge_state, capture's tail
read). standup_recap / transcript_ingest / transcript_chapter are for
MEETINGS and would drag Trello ticket creation and fact-check canvases into
"summarise my tab".
Appendix — everything linked from this entry
- https://docs.propflowai.co/a/gera-tabs-slack-relay
- https://github.com/PropFlow-Technologies/agent-smith/pull/965
- https://github.com/PropFlow-Technologies/agent-smith/pull/966
- https://github.com/PropFlow-Technologies/agent-smith/pull/971
- https://github.com/PropFlow-Technologies/agent-smith/pull/976
- https://github.com/PropFlow-Technologies/agent-smith/pull/977
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README
work on N (messenger mode)
| What it does | `work on 1` in Gera's Slack DM points a DURABLE relay at session #1 of the numbered tab table: every subsequent plain message he types is forwarded into that session through `gera_relay.tell`, and the session's new complete assistant turns are deposited back for the thread's owner to post — and SINCE decision `b91976949` (executed in agent-smith#1085) a 1:1 DM's deposits are read CHAT-WIDE, so the owner of any later line in that DM can be the one that carries them, not only the owner of the root they were filed under; a channel thread is unchanged and still thread-scoped. This runs until he says `stop` or the relay hits its 12h life. `#N` is resolved ONCE, at `begin`, and the record carries the resolved session — so a table he re-draws an hour later cannot silently re-point a live relay. The brain STANDS DOWN for a line the relay is carrying, so one message gets one answer. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | DEPLOYED 2026-10-14 03:38Z. The door is `smith-gera-work-on <N>` (agent_smith.gera_work_on, agent-smith#1012), which the brain runs from prompts.py and which is the ONLY thing that starts the workflow — it starts the relay only on a `begin` that SUCCEEDED, and rolls its own row back if the start fails. The mode is agent_smith.gera_messenger (#968) over ~/.claude/smith-state/gera-messenger-mode.json; the loop is the Temporal workflow SmithGeraMessengerWorkflow (agent_smith.workflows.gera_messenger, #975) on task queue propflow-smith in the Smith worker, ticking every 20s with a 12h bound and continuing-as-new; the SEND is gera_relay.tell (#1008), at-most-once by construction. The stand-down is in dispatcher.dispatch + workflows/reply._work_path (#1016). Reads the session's transcript JSONL for what it says back; deposits through thread_findings and wakes the thread's owner, which is the only actor allowed to post (ADR-0051). |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → work on N (messenger mode) |
Liveness signals
| Signal | How it is read |
|---|---|
| the loop's host is loaded | launchd · co.propflow.smith-worker |
Deployed is not exercised, and the difference matters here. The door merged
at 2026-10-14T03:38:06Z and the worker restarted 40 s later (epoch
1791949086 → 1791949126; both converted before comparing, because ps lstart is local and gh mergedAt is UTC and comparing them raw is how this
fleet got the same question wrong once before). So the code is live. But a
count of SmithGeraMessengerWorkflow executions against Temporal Cloud
(propflow-tools-prod, no status filter) returned 0 at 04:09Z, and
still 0 when re-counted at 2026-10-15T05:50Z — nobody has run it yet,
which is also why the worker log has no gera_messenger lines to build a
signal from. A row that said "active, and here is its usage heartbeat" would
be describing a system that has never run.
Why the loop is a workflow and not a reply turn. The row that asked for
this said it in those words: "Smith's reply turn must not poll or background
anything." A claude -p turn has no execution after its final result, so a
turn that promised to keep watching would simply end —
dm_sessions' own rule, written after a turn promised "I'll keep posting as
they land" and then stopped. The watching lives in a durable run that survives
the turn, the worker restart and the night.
It posts nothing itself. ADR-0051
gives a human's thread one voice — the reply workflow that owns it — and that
gate is live in code, not prose. So the relay DEPOSITS what the session said
and WAKES the owner, and the owner decides whether and when to say it.
⚠️ The DM half of that hand-off was once unreliable and has been REPAIRED,
and the two halves now differ:
workflows/gera_messenger.py retracted its own earlier claim that a finding
"rides the owner's next reply", because in a DM his next line gets a new root
and a new owner, so a finding under the old root was read by nothing. Decision
b91976949 — raised 2026-10-14 directly on the decider's own NEEDS HUMAN
receipt f4e35edfd, which until then had no block at all — was ANSWERED the
same day ("File answers under the whole chat, so they reach Gera") and
EXECUTED in agent-smith 9bbcc334 (#1085). thread_findings.waiting now
ignores thread_ts in a 1:1 DM and reads the WHOLE chat, gated by
gera_tabs.keys_on_the_whole_conversation, so a finding deposited under an
older root IS reached by his next carrying reply anywhere in that DM. ⚠️ Three
things the repair did NOT change, because the row must not be read as a
promise: the DEPOSIT still keys on the root (that root is who gets woken and
what stranded reports a stuck row under); a CHANNEL thread is still scoped to
one thread, where a failed wake still means he may never hear it; and in a DM it
still costs the update outright when no later reply ever happens.
⚠️ This row previously cited b91874375, and that was a citation error, not
just a stale id: b91874375 was about where a DM's #N numbering is stored,
and its own text said "Nothing is blocked on this answer." It is now withdrawn
— its narrow option was already the deployed state (gera_tabs.anchor_of) and
its wider option is agent-smith#1060. A reader following the old citation landed
on a decision that disclaimed being a blocker and concluded this was parked.
Nothing here may claim that a relayed reply reaches him. That sentence
stands, but on a DIFFERENT reason than the one above it used to carry: it is no
longer "a finding under the old root is read by nothing" — it is that DEPOSITING
IS NOT POSTING. Only a send says a thing, this path creates none, and the whole
path has never executed in production (the execution count at the top of this
row). Deposited is not delivered.
And the brain standing down is what makes one message get one answer — ADR-0053, which is where the relation between this system and the reply brain is written down, including the thing that makes it subtle: the stand-down must not skip the DISPATCH, because the run it would skip is the very owner this relay signals its update into.
Appendix — everything linked from this entry
- https://docs.propflowai.co/a/gera-tabs-slack-relay
- https://github.com/PropFlow-Technologies/agent-smith/pull/1008
- https://github.com/PropFlow-Technologies/agent-smith/pull/1012
- docs/adr/0053-a-relay-owns-gera-s-line-and-the-brain-stands-down.md
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README
mini-tab-ledger (which mini sessions the Mac has a tab onto, with history)
| What it does | Every 5 minutes on the MINI, records which of its tmux sessions a Mac-origin client is attached to, with first_seen (seeded from tmux's `client_created`, sticky forever after), last_seen (max client_activity — the last-keystroke clock), present, mac_clients and a pre-computed `order` rank. Writes ~/.claude/mini-tab-ledger.json. Read-only with respect to everything else: `tmux list-sessions` / `list-clients` and `ps`. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/dotclaude |
| Deployed at | launchd co.propflow.mini-tab-ledger (gui domain, StartInterval 300, RunAtLoad true, MINI ONLY) running /Users/miniclaw/.local/share/propflow/mini-tab-ledger/scripts/mini-tab-ledger.py. Logs ~/Library/Logs/mini-tab-ledger.{out,err}.log. Declared in propflowai config/automations.toml under the same id. |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → mini-tab-ledger (which mini sessions the Mac has a tab onto, with history) |
Liveness signals
| Signal | How it is read |
|---|---|
| recorder scheduled | launchd · co.propflow.mini-tab-ledger |
| it is still writing | file-age · ~/.claude/mini-tab-ledger.json · stale after 10 min |
⚠️ PRESENCE IS NOT LIVENESS, and last_seen is what stands in for the half
that is missing. The mini runs sshd with ClientAliveInterval 0, so a client
from the Mac LINGERS in tmux list-clients for hours after the Mac sleeps. That
is useful — it keeps the tab set visible while the Mac is away, which is the
point — but a listed client is not evidence the Mac is awake, and a heartbeat
that advanced last_seen on mere presence would claim exactly that. So
last_seen is max(client_activity): tmux's own last-keystroke clock, which
stops moving when the far end stops typing. It under-reports an open-but-idle tab
and never over-reports a sleeping one. Measured 2026-10-12: of ten Mac-origin
clients, nine had activity inside 9 minutes and one was 4h21m stale while still
fully listed as attached.
A consumer has THREE cases, not two (agent_smith.gera_tabs.read_ledger is
the reference implementation): ok → render; status != "ok" → "could not
read", never "no tabs"; and ok with updated older than ~600s → ALSO "could
not read", because a fresh-looking status over a stale timestamp means this job
is dead and rows that were true an hour ago are not a current answer. The third
is the one a careful reader gets wrong, and it is checked on updated rather
than ok_tick — a live-but-failing recorder writes a fresh updated with a bad
status, so reading ok_tick for staleness calls a broken recorder healthy and a
dead one alive.
A corrupt ledger is NOT rewritten. Overwriting it would turn corruption into
a silent reset of every first_seen — i.e. of the order data this job exists to
accumulate — so the tick reports ledger_unreadable, exits 1, and leaves the
file alone. On any non-ok status no last_seen moves, no row is added or
dropped, and ok_tick keeps naming the last tick that actually read tmux, so a
consumer can always tell how old the data behind a failed tick is.
sid is passed through VERBATIM and its contract says it "may be a full
uuid, a legacy 8-char id, or """ — 8 chars never name a session. A consumer
that resolves or pads it manufactures an identity nothing on the box agrees
with, and tmux is the row's actual primary key.
Appendix — everything linked from this entry
hello-email (hello@ and customer mail → Slack)
| What it does | Every 2 minutes, lists hello@ group mail and mail from mapped customer domains in gera@'s Gmail (read-only) and posts whatever Slack has not seen: one Slack thread per Gmail thread, quoted chains stripped, routed by sender domain to the customer's channel (fallback #hello-email), posted verbatim under Smith's bot with the mirror marker so Smith never treats a customer's email as its own words. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.hello_email from the MAIN checkout ~/code/PropFlow/agent-smith (launchd co.propflow.hello-email, system domain, every 120 s; deploy/co.propflow.hello-email.plist); state ~/.claude/smith-state/hello-email/state.json |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → hello-email (hello@ and customer mail → Slack) |
Liveness signals
| Signal | How it is read |
|---|---|
| poll scheduled | launchd · co.propflow.hello-email |
| listing Gmail | file-age · ~/.claude/smith-state/hello-email/state.json · stale after 15 min |
Why it exists. hello@propflowai.co is a Google Group, not a mailbox. Its
copies land in gera@ under Gmail's Forums category, and Gmail mobile does not
notify on Forums, so customer escalations sent to the group went unseen.
Customers also write straight to a person: of 66 customer-domain emails in 30
days, only 12 went through hello@. Slack is where the team already looks.
It went dark for eight days once. It first ran from a worktree of the unmerged agent-smith #619. When that worktree was swept (~2026-09-25), every tick exited 78 and the channels simply went quiet. A quiet channel reads as "no customer mail today", so nothing looked wrong. It now runs from the main checkout, and the plist test in agent-smith pins that. The poller also posts in #hello-email when its Gmail token dies or a whole run fails.
Appendix — everything linked from this entry
CI_RUNS_ON canary
| What it does | Reads propflowai's CI_RUNS_ON Actions variable every hour and pages #alerts when it is not 1 or cannot be read, re-paging every 6h while it stays that way |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | Temporal tools-prod :: schedule `ci-runs-on-canary` → SmithCiRunsOnCanaryWorkflow on propflow-batch-queue |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → CI_RUNS_ON canary |
Liveness signals
| Signal | How it is read |
|---|---|
| canary ran this hour | log-match-age · ~/Library/Logs/ci-runs-on-canary.log · matches ci-runs-on state= · stale after 180 min |
| flag reads 1 | log-match-age · ~/Library/Logs/ci-runs-on-canary.log · matches ci-runs-on state=on · stale after 180 min |
CI_RUNS_ON picks RunsOn (1) or GitHub-hosted runners (anything else) for
every job in propflowai's ci.yml. On 2026-09-21 it was flipped to 0 and
nothing noticed for a week (~$2K): the flip is one API write, CI stays green,
and the only symptom is a bill. The canary is READ-ONLY by construction — it
never writes the variable — and an unreadable variable (404, auth, network)
pages as "could NOT read", never as healthy. Its ci-runs-on-canary
AutomationRun is on the Jobs tab; its manifest entry is alerting = false so
no autonomous remediation acts on the flag.
What the two signals prove. Each pass appends ONE line to its own
~/Library/Logs/ci-runs-on-canary.log (not smith-worker.err.log — see
phase-snapshot below; its tail window was ~11 minutes until agentflow-relay#343 made
the window widen to the signal's own staleness horizon). canary ran this hour
matches every pass, so it goes stale only when the schedule or worker stops
firing. flag reads 1 matches only an on pass, so it goes stale three hours
into an off or unreadable stretch — the Systems tab then shows the same red
the #alerts page already did. 10800s is three missed hourly fires.
Appendix — everything linked from this entry
row fence
| What it does | Stops a superseded or mistaken lane emitting an external effect on an action item it does not hold, by making every fenced gateway name its item and validating that name against the set the lane actually holds |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | in-process :: agent_smith.row_fence, consulted by smith-pr-drive and smith-task |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → row fence |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
Vocabulary first, because two mechanisms here are one careless sentence away
from sharing a name (ADR-0023
decision 4). A Driver is assigned an action item — since 2026-09-12 setup hands the Driver a dock and the Driver assigns its items to sub-agents; nothing self-claims (ADR-0025 decisions 4 and 9). The verb was claim until then, which is the word ADR-0023 decision 4 records and the word the descriptor on disk still uses. Claim was also the product's
word, and DriverWorkflow.assign (renamed from DriverTaskWorkflow.claim) is the door. Underneath, the thing that
hands out the monotonic fencing generation is the fenced row lease
(SmithRowLeaseWorkflow — the Smith prefix is dropped fleet-wide per ADR-0025 §9, and this type's final name is the one OPEN item on that list, pending Fable, because Astra argues a RETRY is a second writer even with one driver), which keeps the word lease as a named lag and is
not SingleRunnerWorkflow (renamed from SmithWorkClaimWorkflow, ADR-0025 §9): they have opposite lifetimes, which is the
whole reason the generation cannot live in the latter. This entry says assignment
for what a lane takes and fenced row lease in full for what grants it.
What it is. The lane's descriptor (row-fence.json in its task dir)
carries a set — the board plus every action item this session holds right
now, the union of the arming-time row and any mid-life assignments. It answers one
question: what may this caller touch? The gateway then takes --item,
because it genuinely cannot infer one — nothing in a gh pr create says which
of a board's items the PR is for — and validates that --item against the
set. The validation is the mechanism; the flag alone is not. An unvalidated
--item would fence something, just not the thing being worked, and come
back granted: a lease with no fencing generation wearing a different costume.
Four cells, and they are the contract:
| descriptor | --item |
result |
|---|---|---|
| present | in the set | fenced at that item's generation |
| present | not in the set | exit 3 item_not_held |
| present | absent | exit 3 item_unnamed |
| absent | either | UNFENCED, announced on stderr |
Exit 3 is the fence said no; exit 1 is the fence could not be read.
Both stop the effect, and they are kept apart because they demand opposite
reactions — a couldn't read reported as a said no sends a lane to bed when
it should retry.
The board id is validated against KNOWN_BOARDS at both the read and the
write, never free-form. This is not defensiveness: an unvalidated board fails
in the direction that looks like success. portfolio-architecure — one
transposition — resolves to a fenced row lease nobody else will ever ask about,
so every guard is granted, every gateway prints allowed, and the lane writes
with no fence at all. A no-op fence and a working fence are indistinguishable
from the outside. Checking at the write too means the failure names the
publisher rather than the gateway, which is the component that did nothing wrong.
Liveness: why signals is EMPTY, and what would fill it
An empty list renders UNKNOWN, and UNKNOWN is the correct reading today — not a gap somebody should close by adding a signal that goes green.
⚠️ The 2026-09-11 premise under this section has EXPIRED. The conclusion has
not. It read, verbatim: "Nothing has been assigned an action item in
production yet, so the fence has no steady-state emission at all." That was
true when it was written and is no longer true: as of 2026-10-15 one live dock holds
dozens of fenced rows — the published board carries 65 rows whose owner
is that dock, and the dock driver's own sweep passed 75 — the fence refused
a real publish for item_unnamed and allowed it once the item was named, and
fifteen leases lapsed unwatched. (The two row counts are different grains, and
neither is the point: both are far from zero, which is what the expired sentence
asserted.) There is steady-state emission now — precisely the
condition this section was waiting for. It is corrected in place rather than
appended because an expired premise reads exactly like a live one, and the
next reader would otherwise re-reject the three candidates below for a reason
that no longer holds.
Why signals is still empty, on the reason that does hold. A signal that
cannot tell working from never invoked is worse than none, and this catalog
exists to stop exactly that. Emission existing is necessary but not sufficient:
something must emit per guarded write, and nothing does — every failure
mode here is a write that simply was not guarded, and nothing emits when a
guard does not run. None of the three candidates below closes that gap for
this fence: the first measures a different system, the second never resolves,
the third has no fixed path and its absence is also the normal state. So the
three were measured and rejected on 2026-09-11, and the rejections stand on
their own merits:
~/Library/Logs/smith-row-sweep.log— real, and correctly formatted forlog-match-age, but its only line isrow-sweep rollout-x: 3 rowsfrom a test board. It proves the board-owner sweep ran once in a test, and it would go green while the fence was entirely broken: it is a different system. ⚠️ What is rejected here is that log as a signal for THIS fence — not the sweep, and not the log. A different system gets its own entry, and it now has one: seerow-lease-sweepbelow, where that same file is the correct signal because there it measures the system that writes it. Patching the sweep log intorow-fence'ssignalswould be doing the thing this section argues against; adding signals torow-fencemeans finding something the fence emits.- The worker log's
row-lease: published …line — the publish genuinely logs. Butsmith-worker.err.loglines open with ANSI escape bytes and an ISO-8601 stamp (\x1b[2m2026-09-11T07:33:40.016278Z), andlog-match-agerequires the line to start with its stamp. agentflow-relay#349 made the ISO stamp itself readable, so the ANSI prefix is now the whole reason: the^anchor cannot see past it and the line still rendersunknown("matched line has no timestamp") forever — a signal that never resolves is a row that teaches people to ignore the page. - The descriptor file itself —
~/.claude/jobs/tasks/<slug>/row-fence.jsonis a glob, andfile-agedoes a singlestatSyncon an exact path. Its absence is also the normal state for a lane holding nothing.
⚠️ gate printing allowed is NOT evidence the fence is live. This is the
defect that was actually shipped and fixed here: resolve_path returns a path
whether or not a descriptor exists there, so a lane with $SMITH_TASK_DIR set
and nothing published printed row-fence: allowed and read as fenced while
nothing had been asked. An unfenced effect announcing itself as allowed is the
one thing this module must never print. It now says
UNFENCED — no descriptor at <path> for that case, so the audit trail
distinguishes granted from there was no fence to ask. Anyone reading a
transcript for proof the fence worked must look for allowed — … on item <id>
with an item named; allowed with an empty item is the old bug's signature.
What a dead fence looks like: silence. Every failure mode here is a write
that simply was not guarded, and nothing emits when a guard does not run. That
is why the liveness question is answered by whether a descriptor exists for
a lane that holds something — and why the honest signal arrives only once
something is assigned. When a Driver genuinely holds assigned items, the fillable signal is a
file-age on that lane's descriptor, refreshed every heartbeat tick; it needs a
fixed path, which needs the arming path below to be closed first.
Park vs release: when a lane may give a row back, and how an assigner reads it
Park after the last write; release only once the published page reads
merged. smith-row-lease park --reason keeps the row HELD — its own
flag help says the row "is STILL HELD and still has to be heartbeated —
parking buys no time and takes none away" — while recording, in a sentence a
human can act on, that nobody is working it. release is the other verb and
its help is honest about what it is for: "give the row back early." A row
released before the page reads merged reads unclaimed to every
claim-based reader: smith-row-lease sweep classifies it released, and
dock-claimed-items-merged exits 3 — no live claims: absent, empty, or
all RELEASED — so a correctly-finished row drops out of the gate's population
and is offered again. The practice is established and FINISHED on
bug-a-row-can-be-claimed-and-dispatched-while-an-open-pr-already-names-it-so-the-same-work-gets-built-twice;
it is written here because that row is where it was proven and this file is
where an assembler of such lists actually looks.
Why release-after-publish is too early, which is the counter-intuitive half.
Publishing a row's completion is not the end of the row, because the page's
status is derived rather than written: propflow-docs bin/refresh-tracker
computes it from the row's refs, and finisher reads the PUBLISHED dock at a
ref and exits 0 only when every row reads merged or shipped. Between the
write and that derive sits a window in which the work is finished, the row's
own data says so, and the page does not. A lease released inside that window is
released against a page that still reads open.
⚠️ A row with completed_by set is NOT available, whatever its status says
— so an availability filter must test completed_by being empty and never
status. From refresh-tracker's own record: a row finished by TWO refs,
neither of which closed it alone, carries two honest partial entries, and
because merged requires every ref to close in full the finished row reads
open. Three of them were measured on this very board on 2026-09-12 —
bug-dark-field-guard-defeated-by-optional-chaining,
bug-rows-name-their-files-in-prose-so-no-tool-can-read-them and
bug-a6-artifact-and-main-disagree-on-the-field-name — and that file names the
consequence rather than leaving it to be noticed: "an open row stays in the
row-parallelism tool's candidate set, so finished work keeps being offered as
dispatchable."
⚠️ Ownership comes from the LEASE — never from a parallelism result, a status
field, or the absence of a claim. optimizer's verdict says which rows MAY
run at once; it says nothing about who holds them. Measured against
~/.local/bin/work-optimizer at local-bin origin/main on 2026-10-14 (the
checkout 0 behind): main() admits a row on status not in ("open", "wip")
alone, and the file carries zero references to completed_by, holder,
claimant or the lease — its output key is parallel_safe, which is a
statement about disjoint declared paths. Reading that key as availability is
the defect
bug-a-coordinator-reads-work-optimizer-parallelism-as-availability-so-it-assigns-rows-another-driver-holds-and-the-correction-cannot-reach-it
is named for. The authority is smith-row-lease state, or the sweep's holder
field.
⚠️ The descriptor OVER-REPORTS, which is the safe direction and still not the
authority. row-fence.json grows through row_fence.add_entry, which merges
rather than overwrites, and the two release doors do NOT behave alike. The
Temporal row_lease_release_activity drops the item — _drop_from_fence
republishes the remainder and only the LAST release withdraws — but
best-effort: an OSError or ValueError there is logged and never raised,
so the entry stays. The hand smith-row-lease release does not rewrite the
descriptor at all; row_lease_cli.py's only row_fence calls are
is_known_board, hand_sid_refusal and add_entry. The asymmetry is
deliberate and argued at the activity: a leftover item names a generation the
lease has moved past, so the fence refuses it — a confusing refusal — whereas a
missing one "would have read as 'no row' and waved a write through." So read
the descriptor as a claim and smith-row-lease state as the fact.
⚠️ An unreadable answer is not none. sweep carries unreadable as its
own class beside held / lapsed / released / never_claimed, and a pass
that read no row at all exits 1 with "could not tell — every row was
unreadable" rather than reporting zero lapses. state keeps the same two
answers apart on purpose: a genuinely absent lease is exit 0 with
exists: false, while a lease it could not reach raises to exit 1 — because, in
that command's own words, "a caller that could not tell those apart would treat
an outage as an unclaimed row." An assigner that reads either as nobody holds
this assigns a row somebody holds.
Residuals, named — a catalog that hides these is worse than none
item_unnamedwill refuse the first Driver that is genuinely assigned an item.prompts.pyinstructs agents to runsmith-pr-drive addandsmith-task verdictwith no--item, and there is no shell caller of either verb anywhere in the repo — they are instructions to an agent. The moment a lane holds a descriptor those instructions produce exit 3. This fails closed and loud (the refusal ends "Pass --item.") and is unreachable while nothing is assigned, so it is recorded as a decision rather than guessed at.operator-herein~/.local/binstill passes no--board/--row. That is a live shared checkout where any edit is an unreviewed fleet-wide deploy, so the out-of-repo half of the arming path is deliberately untouched.operator-workflow claimis the door that works today (ADR-0025 §9 namesdriver-workflow assign, which is not on PATH, and naming a command that does not exist on PATH is the exact defect thefinisherentry records).- ⚠️ A non-zero exit from an acquire means STATE UNKNOWN, never "nothing
happened". Observed 2026-09-11:
smith-row-lease acquireconnected, took the row, published the descriptor, printed both lines — and then died of SIGSEGV in interpreter finalization, so the caller read 139. The work was complete; the process was not. The general form is worth holding beyond this system: an exit code describes the process, not the work, and the window is widest exactly where the work is most side-effectful, because the world has already changed by the time finalization runs.cli()now flushes andos._exits rather than finalizing, but a caller must still treat a non-zero acquire as go and look, not as it did not happen. - Raw
gh pr createfrom an interactive session is still unfenced.smith-pr-drive addruns after the PR exists.hooks/command_gate.pyis the only surface that could close it and it is wired inline for Smith-brain sessions only, so a human-started session does not run it.
status is proposed, not active, and that is the same honesty as the
empty signal list: the mechanism is built, reviewed and tested, and it has not
yet fenced a single production write. It becomes active when a Driver holds an ASSIGNED action item and
an action item and its descriptor is on disk.
Shipped by agent-smith #452, on top of the fenced row lease (#449) and the Driver assignment door (#453, shipped as the "claim door").
Appendix
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README (no docs declared in SYSTEMS.md yet; add some)
row fence resident client
| What it does | Answers the row fence's one question from a process that is already connected, so a hooked write (git push, gh pr create) never pays the per-invocation Temporal SDK import or a cold connect — caching the CONNECTION and never the verdict |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | BUILT, NOT LOADED — agent-harness #1904, then part 2 in #1908, #1910, #1911, #1912, #1914, #1915 and #1916. Loading the launchd job and pointing guard-bash rule 15 at the asker is one human step (ADR-0069 §6 and its 2026-11-03/04 amendments) |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → row fence resident client |
Liveness signals
| Signal | How it is read |
|---|---|
| resident client loaded | launchd · co.propflow.row-fence-client |
| connected to Temporal | file-age · ~/.claude/jobs/row-fence-client/heartbeat · stale after 2 min |
The catalog's half of ADR-0069.
Two pieces, both in agent-harness: row_fence_resident, the long-lived
process holding one warm Temporal connection, and row_fence_client, the
hook-side asker (stdlib only, no SDK), which talks to it over
~/.claude/jobs/row-fence-client/sock. The resident client answers from the
same row_fence.gate the cold CLI runs.
⚠️ status is proposed and both signals read down today. Both are
correct. The code is built. What does not exist yet is a running process:
the launchd job is not loaded, and rule 15 still runs the cold
row_fence_cli. The signals are declared now because the label and the file
now exist in code, and they are the right signals to read the moment it
loads (ADR-0009):
launchdis the process. KeepAlive restarts it after a crash, and after a deploy, because the process exits on purpose when the package's code changes under it (the asker sends a code stamp with every ask).file-ageonheartbeatis the CONNECTION. The file is written every 30s, and only after a Temporal call on the warm client succeeded. A process that is loaded but cannot reach Temporal ages this file while launchd still reads healthy. That is exactly the state in which every row-bound hooked write is refused "could not ask", so it must show. 120s is four missed beats.
What it is for. row-fence above is consulted by every fenced gateway and,
through guard-bash rule 15, by every hooked write from a row-bound lane. The
gateways are unbounded and reach a verdict by waiting — one measured read
took 85.39s and returned rc=0. The hooked path is bounded at
6.5 − 1.0 = 5.5s and loses a three-way race against its own module import:
measured 2026-10-18 on 14 cores, the import alone costs 3.5–4.7s while the
hook hands the helper 4262–5520ms, and the verdict itself took 20.23s
and came back ALLOWED. A correct answer nobody can wait for. At 1-min load
27 the bound was met 2 of 2; at 32 and above, 0 of 8.
It flips to active once the built client is loaded and has answered one
real fenced write — with signals that can tell a dead client from a missing
one, because an unreachable client must read as could not read (DENY) and
never as not row-bound (ALLOW). Loading it is a human act in its own PR,
the same floor ADR-0058
holds for the lapse sweep.
Appendix — everything linked from this entry
row-lease lapse sweep
| What it does | Classifies every row of a phase board against its lease and names the lapsed ones, so a row that can be neither worked nor adopted is found by a job instead of by whoever happens to look |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | NOT SCHEDULED. `smith-row-lease sweep` exists and is run by hand; the system-domain `fleet-daemon` registration that would give it a cadence is a separate PR and a human load — see ADR-0058 |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → row-lease lapse sweep |
Liveness signals
| Signal | How it is read |
|---|---|
| phase board swept | log-match-age · ~/Library/Logs/smith-row-sweep.log · matches row-sweep portfolio-architecture-phases: · stale after 180 min |
⚠️ status is proposed and this signal reads down today. Both are
correct, and neither is a gap to close by softening them. The tool is built
and works; what does not exist is a schedule. Until the registration
ADR-0058
calls for lands and is loaded by a person, this row means "the fleet has a
lapse detector and nothing invokes it" — a true and useful thing for the
registry to say, and the opposite of coverage. Flipping it to active before a
daemon exists would make the catalog assert a watch nobody is keeping.
What a lapse costs, which is why this row exists. A claimed board row
carries a 3-hour lease (expires_in_seconds 10800). Once it lapses on a
still-open row, that row can be neither worked — the holder's generation is
stale, so its writes are refused — nor adopted, because another lane's
claim is refused while the row is held. On 2026-10-15 a dock driver's own hand
sweep found fifteen rows lapsed, thirteen of them still open, known only
because that driver happened to look — attributed to the sweep that recorded
it, not re-measured here, since the board's leases have moved since.
The detector is sound, and it refuses to self-certify. Run by hand against
75 rows: rc=0, rows_asked 75, rows_read 75, counts
{held: 35, parked: 40}, 0 lapsed, 0 foreign holders, recorded: false. It
writes the liveness line only under --record, because that line is the
evidence the scheduled sweeper ran and a hand-run that wrote one would make a
board nobody armed read as watched. It also keeps clean and read nothing
apart by exit code: a pass that read no row at all exits 1 and says so, while a
pass that read rows and found no lapse exits 0 and says that.
It has swept a real board zero times. smith-row-lease sweep-status --board portfolio-architecture-phases returns rc=3, status "never". The log holds
exactly one line — 2026-09-10 23:50:46 row-sweep rollout-x: 3 rows — held 1, lapsed 1, never_claimed 1 — and rollout-x is a test board. That single
line is also the positive control for the signal above: the format parses, the
writer works, and a real lapsed count has rendered once.
Nothing schedules it. fleet-daemon --status lists 67 labels, 8 of them
out-of-scope, leaving 59 in scope (43 daemon, 12 deferred, 2 agent,
2 disabled). Four are sweeps (ccswitch-walled-sweep, decider-sweep,
decision-block-sweep, inspo-sweep). No row-lease or row-sweep daemon at
any of those grains. Paired control: doctrine-pull is in that listing
(grep count 1) while the target greps 0, so the registry is complete and
greppable and the absence is about this job. ⚠️ An
earlier attempt to prove the same absence grepped ~/Library/LaunchAgents and
found nothing — and its control for doctrine-pull also found nothing
there, while doctrine-pull demonstrably runs. These jobs live in the
system domain through fleet-daemon; a control that fails alongside the
thing it controls has not certified an absence.
Why this signal can tell "never swept" from "swept and clean"
It reads exactly what sweep-status reads. The catalog is read-only by
construction and has no signal type that shells out to a command, so the signal
cannot invoke the subcommand — and does not need to. sweep-status's needle is
" row-sweep <board>: " against ~/Library/Logs/smith-row-sweep.log, and the
log-match-age above names that same file and that same needle. The evaluator
then reproduces the subcommand's three-valued answer:
| this signal renders | because | sweep-status equivalent |
|---|---|---|
ok / stale |
the board's line is there, recent or old | rc 0, status "recorded" |
down |
the log was read and holds no line for this board | rc 3, status "never" — today |
unknown |
the log could not be read at all | rc 1 |
So swept and clean, never swept and could not tell are three different
readings by construction, not by a convention somebody has to maintain.
That is the property the three candidates rejected under row-fence could not
offer, and it is why this system is signalled off its own evidence rather than
off anything the fence emits.
⚠️ The needle is board-scoped, and that is load-bearing. A bare
row-sweep pattern would match the rollout-x test-board line and render
ok for a production board nothing has ever swept — the same defect this entry
documents, one layer down.
staleAfterS is 10800 because that is the lease TTL. A lapse takes three
hours to form, so a sweep older than three hours has not seen the current set.
When the schedule lands, this entry changes in three places and nowhere
else: status → active, deployedAt → the real fleet-daemon
registration, and a second signal on the daemon's own liveness beside the
behavioural one. The path, the pattern and the threshold do not move. Each
additional live board sweeps under its own board-scoped signal, for the same
reason the needle is scoped at all.
What the scheduled sweep may never do: owner-start, acquire, or
heartbeat. It is observation only — it reads row ownership, classifies it and
appends one evidence line. That is already true of the code, and ADR-0058 makes
it a constraint on anything built under it, so the job that watches the leases
can never become a participant in them.
Appendix — everything linked from this entry
Smith daily self-review
| What it does | Every morning Smith reviews everything it said and did since the last checkpoint — sweeps the fail tags, writes every finding as a tracker action item, fans the fixes out as PRs, drives them to merge, self-certifies each completion naming its PR (prod proof in #agent-smith is the encouraged stronger grade), and records the checkpoint |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | Temporal tools-prod :: MorningQueueWorkflow step `smith-review` → SmithTaskWorkflow `smith-goal-<key>` on propflow-smith |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → Smith daily self-review |
Liveness signals
| Signal | How it is read |
|---|---|
| dispatched this morning | log-match-age · ~/.claude/smith-state/self-review/dispatch.log · matches smith-review dispatched · stale after 1800 min |
| checkpoint recorded | file-age · ~/.claude/smith-state/review-checkpoints.db · stale after 1800 min |
The smith-review skill's own preamble calls itself "the inventory that a
future autonomous self-repair engine will run." Until 2026-09-07 every one of
its runs was a human typing /smith-review. Gera: "it should run daily in the
morning queue, and also drive itself to merge." This is that engine
(ADR-0012 — whose "a merge refuses under a person" title clause and Can count Rung 5 prod-evidence clause are both superseded by ADR-0025; the ADR itself is not rewritten):
the morning queue's last step runs agent_smith.self_review run, which decides
the window from the checkpoint store, refuses a second run in one day, posts
the brief as a #agent-smith thread, and starts a durable SmithTaskWorkflow
carrying it on MODEL_HIGH. The review — sweep, items, PRs, merge, self-certification naming the PR (prod proof where reachable),
checkpoint — is the brain's, inside that container. Read the morning's answer
on the phase doc: /a/smith-fail-markers, N/M per group; the receipt is the
thread the brief opened.
proposed, not active, on purpose. agent-smith #426 is open; nothing has
dispatched yet and neither signal path exists on the machine. It flips to
active after the first morning whose thread carries a checkpoint — receipts
before claims, the same rule phase-snapshot below follows.
What the two signals prove, and what they do not (ADR-0009). dispatched
this morning matches the one line self_review writes ONLY when it started a
container — a skipped morning writes smith-review skipped … and a failed
launch dispatch FAILED, neither of which matches, so a queue that stopped
dispatching goes red within 30h rather than reading as quiet. checkpoint
recorded is the mtime of the store smith-review-checkpoint record writes at
Rung 6 — the review reaching its END, whatever it found. A green here does
not mean a clean day; the N/M on the page is where that lives, and record
itself refuses a completion claim over an incomplete sweep. The two can
disagree legitimately: a morning where a human already ran the review is
skipped (signal 1 red-after-30h) with the checkpoint fresh (signal 2 green)
— "the review ran, not by the queue".
Neither signal proves a fix was PROVEN IN PROD — which, since 2026-09-12, is the
STRONGER grade and no longer the bar. The bar is a PR that addresses the point
plus the sub-agent's own completion, which must NAME ITS PR
(ADR-0025
decision 6). That fact lives on the tracker
action item: bin/refresh-tracker (propflow-docs #47, live) derives shipped only from
a merged PR AND a prod_evidence permalink — the STRONGER grade. ⚠️ It is no
longer the done bar. That rule is why the board reads 27 shipped against 114
merged-but-never-exercised; under the 2026-09-12 standard those 114 are done, and
prod_evidence separates a merged fix from a prod-proven one rather than
refusing to count the first. The distinction is superseded, not deleted — it
is kept as the record of why it was drawn, and it earned its keep on the day it
was superseded: a prod preflight read FIVE address collisions where PR #7713's
body had recorded THREE the day before. A PR body is a snapshot of a moment.
Appendix — everything linked from this entry
The Oracle
| What it does | When a Clara turn grades below 8, explains why from the persisted record in the turn's own thread — one plain-English reply naming the cause (prompt gap / missing tool / tool result ignored / policy / grader gap / could not determine) with quoted evidence and no resident detail — and, when the cause has a fix, opens ONE held PR per cause on propflowai; every morning, one line per live property from the persisted grade rows |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | Temporal tools-prod :: OracleTurnWorkflow on propflow-smith (started by propflowai's grading seam) + MorningQueueWorkflow step oracle-nightly |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → The Oracle |
Liveness signals
| Signal | How it is read |
|---|---|
| a low turn was explained | log-match-age · ~/.claude/smith-state/oracle/dispatch.log · matches oracle-turn · stale after 20160 min |
| morning line posted | log-match-age · ~/.claude/smith-state/oracle/dispatch.log · matches oracle-nightly posted · stale after 1800 min |
Not ADR-0086's oracle guard (the human-approval bit on a generated eval scenario). The Oracle is Gera's name for the Agent Smith workflow that explains a low-graded Clara turn — fleet ADR-0014, product propflowai ADR-0132.
Grading (propflowai's postScorecardIfNew, the one seam every detector reports
through) reaches Smith by ONE door: an onLowTurn hook fired after a turn line
lands, when the computed grade is below ORACLE_THRESHOLD (8), starting
OracleTurnWorkflow by type name with id oracle-<conversationId>-<turnKey>
and FAIL on a duplicate. No polling, no cron, no Slack-side listener. The
workflow shells scripts/oracle-evidence.ts from a worktree pinned to
origin/main — the ONLY thing it may quote — and the bucket is decided there,
once, by a table over the record; Smith renders a template (no LLM in the
finding path) and validates it with the live instrument's own predicates
before the send is scheduled. A fix-bearing cause takes a fleet-scoped claim
and starts one SmithReplyWorkflow fix-drive in #agent-smith, which opens
one PR per cause on propflowai, read back. ⚠️ hold-for-review + auto-merge
off was the old shape and is retired (2026-09-12): that PR now merges itself on
green CI plus a cloud-reviewer 🟢, like every other PR in every repo. The Oracle
still does not merge by hand; the PR driver does. The nightly step rides the morning queue (07:00 CT) immediately
before smith-review.
proposed, not active, and the signals are already the real ones. Both
match lines agent_smith.activities.oracle_turn.write_dispatch_line writes
with a local YYYY-MM-DD HH:MM:SS stamp: a low turn was explained matches
oracle-turn <outcome> … — written ONLY when a workflow ran to an outcome
(a refused start in the propflowai seam writes a propflowai log line, never
this one), so it is a per-run pulse and its window is long (14 days: low turns
are sparse). morning line posted matches oracle-nightly posted — written
ONLY when the step posted its lines, so a dead step reads red within 30 hours.
That second signal is the true liveness. Flips to active with no signal edit
once the writer logs (agent-smith PR 2/3 + propflowai #7338 deployed and the
ECS worker provisioned with Smith's Temporal key).
What the Oracle cannot see, said plainly (ADR-0009). A stale injected price with no lookup behind it grades 10 and never reaches it; a pre-field voice call has no persisted briefing, so an unbacked figure there is "could not determine"; a turn the grader never scored is invisible to the morning line, whose footer says so.
Appendix — everything linked from this entry
Driver heartbeat
| What it does | Every 120s, re-derives each live task's phase rows and republishes phases.json plus the thread's pinned one-line /agents pointer — the only ticker keeping the Phases card fresh on a Driver-only task |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/dotclaude |
| Deployed at | ~/.claude/scripts/propflow-status/operator-heartbeat.py |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Driver heartbeat |
Liveness signals
| Signal | How it is read |
|---|---|
| ticker loaded | launchd · co.propflow.operator-heartbeat |
| sweeping | log-match-age · ~/.claude/scripts/propflow-status/state/operator-heartbeat.log · matches sweep scanned= · stale after 30 min |
The LaunchAgent that keeps a running task's PHASE SURFACE current. Every 120s it
walks ~/.claude/jobs/tasks/<slug>/, and for each task with a Slack binding and
a non-terminal state it makes exactly ONE call into agent-smith —
thread_phase_activities.refresh_phases(channel, thread_ts) — which re-derives
the rows once, rewrites phases.json (the file phase-snapshot below
transports), and pushes the thread's one-line /agents pointer at its pinned
anchor.
⚠️ WHY THIS ENTRY EXISTS AT ALL — it is the ADR-0004 case, measured. This
system was absent from this catalog from 2026-08-18 until 2026-09-11, and
that absence is the whole reason the /agents cutover missed it. agent-smith
#371 replaced the thread's eleven-row phase table with one line and a link on
2026-09-04; this script, in a different repo, went on posting the full table
into the same threads. Ledger receipt, 2026-09-07→11: 18 table posts across
7 distinct threads as of 20:27Z on 2026-09-11, the last tick before the
cutover — re-run the query today and it reads 22, because a still-old live
copy of the script re-posted four tables at 20:30–20:33Z, after the first
reclaim and before the final deploy, and the reclaim deleted them again at
20:39:23Z. That is the number to stamp, not a number to quietly update; ADR-0024
records the window and why the second reclaim could fire at all. And on the
2026-09-11 kickoff thread the pointer landed at
20:05:08 and the table 46 seconds later. Nobody checked it because, by this
file's own first sentence, it did not exist. ADR-0024
retires the Slack table and amends ADR-0007.
⚠️ THE SIGNAL IS THE LOG, NOT THE LEDGER, AND THAT IS DELIBERATE (ADR-0009).
The obvious liveness check for this system used to be "did it post to Slack?" —
select count(*) from posts where source='operator-heartbeat' in
~/.claude/smith-state/outbound-ledger.db. After ADR-0024 that query returns
zero forever, by design, so it can no longer tell a working retirement from
a dead LaunchAgent: a signal that cannot fail when the system fails is not a
signal. The tick therefore prints one line per task per tick —
2026-09-11 15:39:35 [operator-heartbeat] <slug> state=<state> refresh=<posted|edited|reminted|unchanged|skipped|failed|declined-…>
for each task it TOUCHED, plus — unconditionally, once per sweep —
2026-09-11 15:59:02 [operator-heartbeat] sweep scanned=3168 bound=11 published=2 declined=0 skipped=0 failed=0 errors=0 unreclaimable=7.
⚠️ The catalog keys on sweep scanned=, NOT on the per-task line and NOT on
the [operator-heartbeat] prefix, and both alternatives are wrong for reasons
worth keeping. A per-task line fires only when a bound, non-dormant task
exists, so a healthy ticker whose live tasks have all finished prints nothing
and this signal reads down. ⚠️ This is the one home for that count, because
it is a LIVE number and not a constant — as of 2026-09-11 20:40Z, exactly
two tasks were bound and non-terminal (out of 11 bound and ~3,169 scanned),
i.e. one finished task away. It is a dated snapshot, not a standing fact: bind a
third task and it changes. ADR-0024 makes the same argument and deliberately
does NOT restate the figure — it points here, the same way it points here for
staleAfterS. And the tick-failure line also carries the
[operator-heartbeat] prefix, so a prefix-keyed pattern would go green
against a log containing nothing but errors — the inversion catalog.ts's own
header warns about. The sweep line is written even when nothing needed doing,
which is the only shape that separates quiet because idle from silent because
dead, and it carries the counts that say which. staleAfterS is 1800s —
fifteen missed 120s sweeps (StartInterval 120 in
co.propflow.operator-heartbeat.plist; 1800/120 = 15, not two). It is set wide
rather than tight on purpose: this line, unlike the per-task one, has no
legitimate reason to be absent, so the only thing 1800s buys is room for a slow
sweep — each one walks ~3,200 task dirs and imports agent-smith — without a
single long tick flipping the catalog to down. The launchd signal beside it
is the fast catch for a job that stopped being LOADED; this one catches a loaded
job that stopped SWEEPING. ⚠️ A LEADING stamp is REQUIRED, not cosmetic —
evalLogMatchAge ages the matched line from a stamp anchored at its START, so a
stamp anywhere else renders this signal unknown forever. This script's first
draft had that bug (fixed in dotclaude#45 before deploy, with a both-directions
control). What is NO LONGER part of it: the stamp does not have to be the
space-separated local form. agentflow-relay#349 widened LOG_STAMP to the
…T…Z and offset shapes after ten rows across the catalog — eight of them over
~/.claude/jobs/specialists.log — were found permanently unknown for exactly
that reason, so LOG_STAMP in relay/src/systems/catalog.ts is the authoritative statement of what the evaluator can date and
no entry restates the regex. The sibling sightings read differently now:
fleet-deploy (ISO instead of local) is repaired by the widening, row-fence
(ANSI bytes BEFORE the stamp) is not.
What it deliberately does NOT do. It derives no phase and owns no Slack
message. Both would be a second answer to a question phase_table.py already
answers once — the failure phase-snapshot below was built to end. It also
declines rather than publishes when a thread is bound to a different task's
slug (thread_phase.claim_task_slug, compare-and-write), because filing one
task's rows under another task's pointer is the one thing a single anchor must
never do.
Appendix — everything linked from this entry
Phase snapshot transport
| What it does | Carries Agent Smith's already-derived phase table to the /agents board as a versioned per-task JSON sidecar — one resolver in Python, two surfaces |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | ~/.claude/jobs/tasks/<slug>/phases.json |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Phase snapshot transport |
Liveness signals
| Signal | How it is read |
|---|---|
| writer loaded | launchd · co.propflow.smith-worker |
| phase evidence recording | file-age · ~/.claude/smith-state/thread_phase.json · stale after 1440 min |
Two systems, one document. Agent Smith (phase_snapshot.py) derives the
table once — in the process that owns the facts — and writes it beside
task.json; this relay (digest/phases.ts) transports it as the additive
phases / phasesUnreadable fields on task rows the feed already serves, and
propflowai renders it on /agents/[sessionId]. The board never computes a row.
ADR-0007 is why the
transport was forced rather than chosen: grep -rn SMITH_STATE_DIR relay/src supervisor/src returns nothing, so half the phase inputs live in a
store nothing on this side can reach.
active since 2026-09-11, and this entry carried the stale claim that made
the flip necessary. It read "nothing is deployed, and no phases.json exists
on the machine yet" — written while agent-smith #348 and agentflow-relay #83
were open — and stayed that way after both merged. Receipts, measured while
writing ADR-0024:
~/.claude/jobs/tasks/arch-design-stress-test/phases.json and
…/phase-pointer-title/phases.json both exist and were rewritten at
2026-09-11T15:39:22 and 15:39:35 local. The flip is late because a proposed
entry that has silently gone live reads as nobody has built this, which is
how a second session builds it again — the disease this file's own header
names. Found by this repo's reviewer on the PR that added the
operator-heartbeat entry above, which measures the very file this entry said
did not exist.
What the signals prove, and what they do not. writer loaded is the
process that does the writing. phase evidence recording watches
thread_phase.json — Smith's evidence store, the very one under
SMITH_STATE_DIR that the board cannot reach — whose mtime ticks on every
phase record; a Smith that has stopped seeing threads for a day stops looking
healthy.
Neither proves a snapshot file landed, and that gap is a limit of the
catalog's vocabulary, not an oversight. file-age takes a fixed path and this
document's path is per-slug (~/.claude/jobs/tasks/<slug>/phases.json), so the
per-task file cannot be declared. The end-to-end proof is a task row on
/api/feed carrying a phases object; a signal type that can watch a
newest-match across a directory is the follow-up that would let this entry
assert it.
⚠️ A log-match-age on ~/Library/Logs/smith-worker.err.log was the first
draft and is wrong, measured rather than guessed: that log grows ~6 KB/min,
evalLogMatchAge tailed only the last 64 KB, so the window it could see was
~11 minutes regardless of the staleAfterS declared. The signal read ok
at 14:29 and down at 15:00 with nothing about the system having changed — it
would have been reporting log volume while claiming to report liveness.
THE WINDOW HALF IS FIXED AT THE SOURCE (agentflow-relay#343, after the same
defect was found holding spend-alarm red every day): the tail now widens from
64 KB until the pattern is found or the window reaches back past this signal's
own staleAfterS, so a declared budget is now honoured rather than silently
clipped to log volume. The rest of this paragraph still stands and is why this
entry keeps its own log anyway: the whole point of a canary is that it fails
differently from the thing it watches.
Appendix — everything linked from this entry
Phase dock rotation
| What it does | Keeps docs.propflowai.co/a/portfolio-architecture-phases true without anyone editing it — re-derives every tracker action item's status from live PR, decision and prod-evidence state (a merged PR naming the item is done; prod evidence grades it stronger — ADR-0025 decision 6), rewrites the dock's live block (running lanes, Astra, decisions taken) and its top 'Waiting on you' queue from the decisions ledger, and ships by pushing when and only when something moved |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflow-docs |
| Deployed at | propflow-docs/bin/refresh-trackers (launchd co.propflow.tracker-refresh, StartInterval 300) → bin/refresh-tracker · bin/derive-live-lanes · bin/sync-dock-phase00 · bin/dock-ping |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Phase dock rotation |
Liveness signals
| Signal | How it is read |
|---|---|
| rotation scheduled | launchd · co.propflow.tracker-refresh |
| rotation ticking | file-age · ~/Library/Logs/tracker-refresh.out.log · stale after 15 min |
| live deriver running | file-age · ~/Library/Caches/propflow-docs/live-lanes-state.json · stale after 15 min |
| dock fresh | file-age · ~/Library/Caches/propflow-docs/dock-fresh.marker · stale after 30 min |
Why this entry exists (2026-09-10). It did not, and by this file's own opening rule that meant it did not exist — while it had quietly become the surface Gera reads most. His words that day: "If something's waiting for me, it needs to be a clear decision link on that portfolio architecture phase doc. At the very top... And I will go and answer those as they come in because I'm looking at that phase doc all the time." The dock now carries a Waiting on you block whose membership is the decisions ledger itself and whose answeredness is re-read each tick from the canonical answer store, so this rotation stopping no longer just freezes a status table — it strands the queue of questions a human is waiting to answer, on the page where he expects to find them. Nothing was watching the watcher.
Why the liveness signals are FILE AGES and not a log match. The terminal
line this rotation prints is 2026-09-10T20:06:07Z all trackers current — nothing to publish, and its stamp is ISO 8601. log-match-age needs
YYYY-MM-DD HH:MM:SS local; handed an ISO stamp it rendered UNKNOWN while
every test stayed green — a signal that reports nothing while looking installed.
⚠️ agentflow-relay#349 made that reason false: the …T…Z form is read now.
The signal is still not a log match, on the reason that outlives the parser —
that line is the terminal line of ONE path, so it is absent on a rotation that
DID publish, and absent must not read as dead.
So liveness is taken from mtimes instead, and deliberately from two different
layers so they fail apart: tracker-refresh.out.log proves the launchd
wrapper woke and ran, live-lanes-state.json proves derive-live-lanes itself
completed, because that cache is written on every derive whether or not the page
changed. Both are set to three intervals (900s), the same "three intervals with
nothing looking" threshold the page renders to its reader.
What these signals do NOT prove, stated so nobody assumes it. They do not
prove the dock is fresh, only that the rotation is running. The published
page's generatedAt is the last tick that CHANGED something — refresh-trackers
neutralises it before diffing so an unchanged page is not re-committed or
re-deployed — so an old stamp on a quiet afternoon is correct, not a fault. Nor
do they prove a deploy: shipping is the push, and Cloudflare Pages can queue
behind other sessions' pushes for many minutes. The end-to-end proof is
bin/docs-fetch /a/portfolio-architecture-phases returning the change.
⚠️ dock fresh is the only one of these four that answers the question a
reader actually has — added 2026-09-11, and the three above it are kept
precisely because they fail differently. On 2026-09-11 a docs push landed inside
this job's four-minute pull→push window, its own push was rejected, the checkout
was left ahead 1, behind 1, and every later tick's git pull --ff-only refused
to advance a diverged branch: 10 rejected pushes, 32 frozen ticks, worst case
45 minutes of a stale dock. All three signals above stayed GREEN for the
whole of it, correctly — the rotation was running, and was faithfully writing
pull failed — not touching the site into that log every five minutes. Green
meant the thermometer worked. That is ADR-0009
stated as an incident.
So bin/refresh-trackers touches dock-fresh.marker on one condition and no
other: the dock was derived on that tick (not skipped for being dirty, not frozen
behind a failed pull) and the commit it was derived against is origin/main.
Nothing else touches it — not a tick that ran, not a tick that logged, not a tick
that failed politely. So its mtime is not "the job ran"; it is the last moment
the published page was what the rotation would have produced, and one signal now
covers a dead job, a wedged repo and a dock left dirty by a co-tenant. The
sibling dock-fresh.json carries the reason in prose and is deliberately not
the signal: it is rewritten every tick, fresh or not, so its own mtime lies.
staleAfterS is 1800 — six intervals, half the worst observed freeze and three
times the "nothing has looked" threshold the page renders to its reader, which is
ADR-0009's corollary
about picking a threshold that cannot flap.
⚠️ AND A RED SIGNAL HERE REACHES NOBODY ON ITS OWN. relay/src/systems/catalog.ts
is read-only by construction and the only importer is relay/src/index.ts, which
serves the /agents Systems tab; nothing under supervisor/src/ reads this file at
all. For a failure whose entire symptom is the page you are looking at has stopped
telling the truth, a signal you must already be looking at is circular. So the
marker is consumed by a second rung — supervisor/src/dockfresh.ts, which stats
the same file each sweep and raises a dock-frozen alarm through escalate()
(delivered to the architect, or an open row on the decisions ledger; never a log
line with no reader). ⚠️ That second rung was the Supervisor, deleted on
2026-10-18 (ADR-0067), so a stale
marker again reaches nobody on its own: only this entry on the Systems tab shows
it. Signal, not call —
ADR-0003: that phase never
runs bin/docs, never takes the docs publish lock, and only ever stats. The two
instruments split the failure modes on purpose: the alarm covers the marker going
stale, and the catalog covers it being absent (a missing path renders
unknown, and unknown never renders healthy), which the alarm deliberately will
not raise on.
⚠️ The per-file dirty guard is what makes this survivable in a shared
checkout. propflow-docs is live and many sessions commit into it at once. The
rotation computes the set of files it would write, intersects that with the
dirty set, and skips precisely those by name rather than bailing on a dirty
tree — because it used to be git status --porcelain → exit, and on 2026-09-10
one foreign uncommitted file stopped all twelve trackers for hours, once every
fifteen minutes, saying so in a line nobody read. A session that leaves the dock
dirty across a tick boundary still silently un-watches that one page, so: commit
what you touch, and leave the tree clean.
Appendix — everything linked from this entry
doctrine pull
| What it does | Reflex — fast-forwards the fleet's live shared checkouts so a merged change reaches every reader without a human pulling it: ~/.local/bin (carrying canon/operator-SKILL.md), the propflowai main checkout, and propflow-docs. Three launchd jobs, one script. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/doctrine-pull, three launchd jobs every 300 s: co.propflow.doctrine-pull (~/.local/bin), co.propflow.propflowai-main-pull (propflowai main checkout), co.propflow.propflow-docs-pull (propflow-docs) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → doctrine pull |
Liveness signals
| Signal | How it is read |
|---|---|
| job loaded | launchd · co.propflow.doctrine-pull |
| tick completing | log-match-age · ~/.claude/jobs/doctrine-pull/doctrine-pull.log · matches health=ok · stale after 30 min |
| propflowai main pull loaded | launchd · co.propflow.propflowai-main-pull |
| propflowai main pull completing | log-match-age · ~/.claude/jobs/propflowai-main-pull/propflowai-main-pull.log · matches health=ok · stale after 30 min |
| propflow-docs pull loaded | launchd · co.propflow.propflow-docs-pull |
| propflow-docs pull completing | log-match-age · ~/.claude/jobs/propflow-docs-pull/propflow-docs-pull.log · matches health=ok · stale after 30 min |
Detail document: ~/.claude/jobs/doctrine-pull/last.json (the Systems tab opens this row into it).
The hazard it closes. ~/.local/bin/canon/operator-SKILL.md is the Driver
manual, and agentflow-relay ADR-0021 §4 makes it a live shared checkout on
PATH for every session, deliberately: merging a doctrine change is meant to
be an instant fleet-wide deploy. That design has exactly one moving part —
something has to pull — and until 2026-09-11 nothing did. Every advance in
that repo's reflog was a hand-run git pull, and local-bin's own README said so
(main here "only moves forward by git pull after a PR merges") without anyone
reading it as a gap. It cost the largest miss of 2026-09-10: the claim commands
(operator-workflow claim/claims/release) existed for hours before any manual in
the checkout named them, and the merge documenting them had to be waited on by
a human before a Driver could read its own arms.
⚠️ co.propflow.puller is not this, and the name is why the gap survived.
That job reads PR states, check conclusions, review verdicts and the decision
queue. It contains no git pull and never touches ~/.local/bin. It was loaded,
healthy and ticking the whole time the doctrine checkout was going unfed — a
green light for a different question.
⚠️ THE SIGNAL THAT LOOKS RIGHT AND IS A TRAP: "the checkout is current." It
was current the night this was investigated — 0 commits behind, both #127 and
#128 present — because a human had pulled it 3 hours earlier. Currency is an
outcome someone may have produced by hand; it says nothing about whether anything
will produce it next time. That is why the launchd signal sits beside the
log signal: one proves something is scheduled, the other proves ticks complete.
doctrine-pull --status makes the same distinction for a human, and reports
JOB_DEAD over a perfectly current checkout when nothing has ticked.
⚠️ AND THE READ-OUT EVERY SESSION ACTUALLY SEES USED TO HIDE THIS.
loop-doctor --version printed a bare short sha of this checkout — which reads
as a version and proves nothing, because the question that matters here is not
"which commit" but "how far behind". Measured on 2026-09-12: the checkout sat 1–2
commits behind for 25 minutes (17:57–18:22, untracked astra-relaunch*
copies blocking the fast-forward, BEHIND_REFUSED health=bad logged six times)
and --version printed a sha and looked healthy for every one of those minutes.
It was resolved by a hand merge — the reflog reads merge origin/main: Fast-forward at 18:26, not doctrine-pull:.
Since local-bin#168 the count is welded to the same line as the sha, so neither can be quoted without the other:
loop-doctor a2214a1 · 6/6 parts linked … · current with origin/main (fetched 79s ago)
loop-doctor a2214a1 · 6/6 parts linked … · 2 BEHIND origin/main (fetched 61s ago)
It does not fetch, and that is why the count can read UNKNOWN. This job is
the only fetcher; a second one would re-inherit the keychain hang #131 fixed, and
--version must stay side-effect-free. So --version reads a LOCAL
origin/main — which is only as fresh as this job — and pairs the count with
last.json's tick age, downgrading to FRESHNESS UNKNOWN past 1800s or with no
tick on disk. If doctrine-pull dies, 0 behind would otherwise be a lie told
in the same confident voice as the truth. That makes --version a third reader
of this job's liveness, beside the two signals above.
⚠️ A WEDGE IS WORSE THAN A FAILURE, and this job found that out in production
on its first tick. It ran ~90s without exiting: state = running, runs = 1,
no status file, no log line, ps showing git-remote-https. launchd does not
start the next tick while one is running, so one wedge is permanent silence —
and it leaves the previous success sitting in last.json looking like health.
The cause was never discriminated and this entry does not pretend otherwise:
the repo's credential.helper = osxkeychain cannot prompt from a background
LaunchAgent, and the machine was at load average ~100 with the job at
Background priority — the ticks that later SUCCEEDED took 62s and 105s, so ~90s
may simply have been an unfinished slow fetch. No git-credential-osxkeychain
process was ever observed. Both are closed: GIT_TERMINAL_PROMPT=0, a
per-invocation -c credential.helper='!gh auth git-credential' (gh's token is a
plain file, needing no keychain and no GUI session), a 240s timeout sized
against those measured 62s/105s ticks, and no ProcessType — Background asks
launchd to defer under exactly the load a fan-out night produces, which would
starve the doctrine deploy precisely when the fleet is busiest. The tick also
writes RUNNING before the fetch, so a wedge renders as HUNG rather than as
the last good tick.
Why the log signal matches health=ok. log-match-age matches with
String.includes — a substring, not a regex (relay/src/systems/catalog.ts,
l.includes(sig.pattern)) — so it cannot say "CURRENT or ADVANCED". A signal on
state=CURRENT alone goes stale after a run of consecutive ADVANCED ticks,
which is exactly what a busy merge night produces: local-bin merged 7 PRs in one
hour on 2026-09-11, and the catalog would have called a perfectly working job
dead. Matching any line is worse — a wedged tick writes RUNNING forever. So
the job stamps health=ok on precisely the two healthy terminal states.
It is deliberately timid, because the directory is on PATH live. Never
reset, checkout, stash or clean; merge --ff-only refuses rather than
clobbers, which matters because ~/.local/bin carries untracked tools nobody has
committed yet. It refuses to act at all off main: a pull --ff-only on a
parked feature branch fails silently, which is how the 2026-06-13 false
spine-health alarm happened.
How you can tell a person did not do it. The job sets
GIT_REFLOG_ACTION=doctrine-pull, so an advance it made reads
doctrine-pull: Fast-forward in the reflog where every hand-run one reads
pull --ff-only: Fast-forward. "Mechanism or person?" is answerable from disk
rather than from a report.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
runtime refresh
| What it does | Every six hours Agent Smith fast-forwards the mini's runtime checkouts — the propflowai main checkout and every repo under ~/code via git-hygiene, then ~/.claude through its own updater — and never applies anything over local-only commits |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | Temporal tools-prod :: Schedules runtime-refresh (01:00/13:00/19:00 America/Chicago, full pass) and runtime-refresh-dotclaude (07:00, dotclaude_only=True) → SmithRuntimeRefreshWorkflow on propflow-batch-queue; the repos' 07:00 pull is MorningQueueWorkflow step 0 (morning-git-hygiene) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → runtime refresh |
Liveness signals
| Signal | How it is read |
|---|---|
| pass completing | file-age · ~/.claude/jobs/runtime-refresh/last.json · stale after 420 min |
Detail document: ~/.claude/jobs/runtime-refresh/last.json (the Systems tab opens this row into it).
Who owns it, in Gera's words (block b90932551, 2026-10-02): "Agent Smith,
which is a local kind of owner of the Mini. And they do have a daily pool, like
in the mornings. But maybe it should be every, like, six hours. So it tries to
stay fresh." The daily pull he meant is morning-git-hygiene, step 0 of the
07:00 morning queue; it still pulls the repos at 07:00. Two Schedules run the
same workflow: runtime-refresh (01/13/19) does the full pass, and
runtime-refresh-dotclaude (07:00) does ~/.claude only, because the morning
queue never runs its updater. Both write the receipt below (slot is full
or dotclaude-only; the 07:00 one carries repos: []), so on schedule it is
written at 01/07/13/19, six hours apart. tests/test_register_runtime_refresh_schedule.py in agent-smith
pins both crons.
The signal is this job's own receipt, not the updater's. ~/.claude's
updater also runs at every SessionStart and writes
~/.claude/jobs/runtime-update/latest.json, so that file's age proves nothing
about Smith. Only a Smith pass writes jobs/runtime-refresh/last.json; it
carries the git-hygiene summary and the updater's status (current,
attention, skipped when a session held the updater lock, or unknown when
the updater could not run). 25200s is one six-hour gap plus an hour. Until the
schedule is unpaused after the worker deploy, this row reads stale — truthfully.
attention is the updater working, not failing. With local-only commits
present it applies nothing and says so; reconciling that local work is a
human's call (row bug-runtime-checkouts-have-no-puller), not this job's.
Not covered here, deliberately: ~/.local/bin, propflow-docs,
ccswitch and the propflowai main checkout also have doctrine-pull
instances on a 300s launchd tick (see doctrine-pull above); this job is the
six-hourly floor under all of them, plus the only scheduled owner of
~/.claude. Re-pinning ~/.claude's nested-checkout gitlinks is NOT done by
any refresh — the updater refuses incoming changes that touch them.
Appendix
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README (no docs declared in SYSTEMS.md yet; add some)
runtime checkouts
| What it does | The inventory of the git checkouts the mini's launchd jobs, PATH and symlinks exec or read at runtime, with one liveness signal per periodically refreshed checkout saying its owner is still running and completing |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/runtime-checkouts (the census, run by hand: plain, --check, --drift, --deploy, --json); its controls 56/60/61/65 run in the weekly fleet-controls job |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → runtime checkouts |
Liveness signals
| Signal | How it is read |
|---|---|
| ~/.local/bin (doctrine-pull) | log-match-age · ~/.claude/jobs/doctrine-pull/doctrine-pull.log · matches health=ok · stale after 30 min |
| ~/code/ccswitch (ccswitch-source-pull) | log-match-age · ~/.claude/jobs/ccswitch-source-pull/ccswitch-source-pull.log · matches health=ok · stale after 30 min |
| propflow-docs (propflow-docs-pull) | log-match-age · ~/.claude/jobs/propflow-docs-pull/propflow-docs-pull.log · matches health=ok · stale after 30 min |
| propflowai main checkout (propflowai-main-pull) | log-match-age · ~/.claude/jobs/propflowai-main-pull/propflowai-main-pull.log · matches health=ok · stale after 30 min |
| agentflow live branch (relay-deploy) | log-match-age · ~/.claude/jobs/relay-deploy/relay-deploy.log · matches SWEEP OK · stale after 45 min |
| ~/.claude (runtime-refresh) | file-age · ~/.claude/jobs/runtime-refresh/last.json · stale after 420 min |
What this entry is for. Fleet tools do not run from a package. They run
from working copies on this machine, and a merge reaches none of them by
itself. runtime-checkouts is the census that says which copies exist and
which job keeps each one current. It pulls nothing. The signals above prove
less than "every checkout is current": each one says its OWNER is still
running and completing, one per owned checkout, and the row goes stale or
down once an owner has stopped for longer than its threshold. What an owner
concluded is in its own receipt, not here: relay-deploy logs SWEEP OK HELD
when it deliberately leaves code commits undeployed, and runtime-refresh
writes its receipt on an attention pass too. Both read healthy above.
⚠️ The census and this entry disagree about three owners, and this entry is
the corrected map. runtime-checkouts recognises only doctrine-pull
instances as owners, so it still reports agentflow, agent-smith and ~/.claude
as unowned. Their owners are named below. Use the census for the list of
checkouts. Use this entry for who owns them until the census learns these
owners.
Owners, and where each is catalogued. Four checkouts are doctrine-pull
instances on a 300s tick (same script, different DOCTRINE_REPO; see
doctrine-pull above): ~/.local/bin, ~/code/ccswitch (with
DOCTRINE_AFTER_PULL=ccswitch-deploy), ~/code/PropFlow/propflow-docs and
~/code/PropFlow/propflowai. ~/code/PropFlow/agentflow is advanced by
relay-deploy (above), which fetches and fast-forwards live and holds any
change that needs a build. ~/code/PropFlow/agent-smith is deployed on every
push to its main by review-webhook running ~/.claude/scripts/smith-sync-from-main.sh
(fetch, reset to origin/main, uv sync --frozen, restart). That is the venv
deploy step this row asked for. It is event-driven, so no age threshold fits
it and it carries no signal here; its receipt is the deploy.done line in
~/.claude/services/review-webhook/listener.log. ~/.claude is owned by
runtime-refresh, whose schedules are registered paused; until they run, that
signal reads unknown, which is true.
The doctrine-pull instances run as system LaunchDaemons
(/Library/LaunchDaemons/co.propflow.*-pull.plist). The relay's launchd
signal runs launchctl list <label> in the user domain, which cannot see
them (exit 113), so this entry uses their log lines only.
Not owned, 2026-10-04 reading. ~/code/Mofu (has a .venv, reached by
co.propflow.mofu-bot): the next step is one more doctrine-pull instance.
Three are a person's call, not a puller's. ~/code/PropFlow/whatsapp-mcp
tracks a third-party origin and carries local commits, so a puller would
deploy a stranger's commits into running bridges; fork and re-point it first.
~/openclaw's origin no longer answers. ~/bin has no remote. Agent Smith's
morning git_hygiene is a daily floor under some of these. It scans one level
below ~/code and ~/code/PropFlow, takes only clones that have an origin
(minus its exclusions), and tries to fast-forward each one's current tracked
branch. That covers Mofu. whatsapp-mcp is in its scan too, and it tracks the
third-party upstream. While the local commits are only on this machine, any
new upstream commit makes the branch diverge and the fast-forward refuses. It does not reach ~/openclaw, ~/bin or ~/.claude.
Also runtime, and self-refreshing. The ~/.claude/scripts/*/repo-main
propflowai worktrees used by the daily scripts are refreshed at the start of
every run by scripts/lib/fresh-main-worktree.sh. It tries to fetch and reset
to origin/main, so they need no puller. It is best-effort, though. A failed
fetch or reset is swallowed and can leave the worktree stale. If no valid
worktree can be set up, it returns the primary checkout instead.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
slug owner
| What it does | Answers whose task slug a register belongs to, so two live incarnations of one Driver cannot both write the claim register and the acceptance gate while each reads the other's lines as its own — and refuses the arm that would create the second writer |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/slug-owner (the judge) :: ~/.claude/scripts/operator-here (the refusal), symlinked onto PATH as ~/.local/bin/operator-here |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → slug owner |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
The hazard it closes. A Driver's task slug (sess-<name>) names a
DIRECTORY, and that directory holds the claim register (claimed-items.txt),
the acceptance gate (acceptance.json) and the row fence (row-fence.json).
The slug is derived from the roster NAME, so a respawned pane lands on the
same slug — which is the design: a dying pane must not lose the work. The cost
is that nothing in the path could tell two incarnations apart.
2026-09-12: a Driver pane respawned, the pre-respawn session did not die
— lanes caught it writing files 6 and 9 minutes later — and four action-item claims
plus three acceptance items appeared under the successor's slug that it never
authored. It asked twice who wrote them before the tmux pane address settled
it: driver-v2, propflow-b4 and propflow-1f all resolved to ONE pane.
⚠️ Nothing was at risk in that instance, and the record must keep saying so. Both writes were APPENDS, and appending only makes "done" harder to reach. The direction was luck, not design — the same seam lets one incarnation release a row the other's lane is still working, which the row fence cannot catch because both are the same holder.
What it is, in two halves. slug-owner <slug> is a judge that writes
nothing: it reads the operatorSid that already binds a session to a slug and
exits 0 MINE, 9 NOT MINE (naming the owner and whether it reads LIVE /
DEAD / UNKNOWN) or 5 CANNOT TELL. operator-here is the refusal: before
binding a session to a slug it consults the judge, and a different owner that
reads LIVE — or whose liveness could not be MEASURED — refuses the arm with
exit 9 and writes nothing.
⚠️ RESPAWN IS NORMAL AND THIS DOES NOT FORBID IT. A predecessor that reads
DEAD is taken over with no flag; --supersede is the deliberate door through a
live one. Either way the takeover is recorded (priorOperatorSid,
operatorEpoch, a bounded handovers list), because a register that names only
a slug cannot answer "who wrote this".
⚠️ 5 IS NOT 9, AND unknown IS NOT dead. Both refusals are non-zero and
demand opposite repairs — 9 is stop writing, 5 is arm it. Liveness comes
from a process list and a transcript mtime across both projects roots
(~/.claude/projects and every ccswitch/session-cfg/*/projects), deduped by
realpath and filtered by isdir, the same shape find-pane uses. That
filter is load-bearing: an absent root globs empty exactly like a present empty
one, so without it "I could not look" renders as "I looked and it was gone"
— and dead is a successor's licence to take a live slug.
What it does NOT close, named rather than hidden:
- Per-line authorship is now answered for the registers that have a writer:
claim lines are stamped and judged by
claim-author(local-bin#220), and acceptance items carryaddedBy/addedAtwritten bydriver-here(dotclaude#135) and bystart-driverat launch (local-bin#335). A line appended by hand with no stamp still reads unattributed — never as yours. - The pending window. Every spawn writes the Driver's sid empty until it
is resolved.
start-operatorresolves it itself for up to a minute; past that, the next Supervisor sweep used to fill it in, and since ADR-0067 deleted the Supervisor nothing does. The spawner now recordsdriverPane, and with no sid the judge asks that pane who is stamped on it now (the stamp, never the pane's name — the 2026-09-12 incarnations shared one pane). The Supervisor carrieddriverPaneand the handover record throughreadTasks(until ADR-0067); before that, its next write erased them, which is why nohandoversentry had ever survived andsignalsbelow stayed empty. A pane name is reusable, so each spawn also sets a random@spawn_nonceon its tmux session and records it asspawnNonce(removed by any write that fills in the sid). The judge reads the nonce and the stamp in one tmux call and answers 5 unless the nonce matches (ADR-0038, amendment of 2026-10-02). - The release half (a sibling freeing a row a live lane is still working) is agent-smith #511, separate and open.
- An absent judge is not a refusal.
operator-hereproceeds and prints⚠ fence: NOT RUN … do not read this arming as fenced— a box without local-bin must still be able to arm, and an unrun fence that read as a pass would be the defect one layer up.
Liveness: why signals is EMPTY, and what would fill it
An empty list renders UNKNOWN, which is the correct reading: at the time of
writing both PRs are OPEN and the refusal has never fired in production. What
would fill it is the refusal's own exit-9 line — operator-here writing a
task-event when it refuses an arm, and the handovers entries accumulating in
task.json, both of which are files a sweep can count. Neither exists yet, and
inventing a signal that nothing emits is how a catalog starts lying.
status is proposed, not active. It becomes active the first time an
arm is refused, or the first handovers entry is written by a real respawn.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
lane scratch namespace
| What it does | Gives every lane of a Driver its OWN subdirectory of the one shared session scratchpad, and refuses a write to the shared ROOT while lanes are live — so two lanes picking the same natural filename cannot silently overwrite each other's work |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/lane-scratch :: ~/.local/bin/canon/guard-lane-scratch.py, linked to ~/.claude/hooks/ and registered under PreToolUse by loop-doctor |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → lane scratch namespace |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
The hazard it closes. A Driver's lanes are sub-agents of ONE session
(operator-SKILL §0d), so they do not get their own scratchpad — they all get the
Driver's, and nothing namespaced it. 2026-09-12: the #7708 lane wrote
pr-body.md at the root; the #7744 lane picked the same obvious name and pushed
#7708's content over #7744's PR description. pr-body.md, v1.md, v2.md,
v3.md and v4.md all sat unprefixed at that root with eight lanes live.
⚠️ Worse than an ordinary race, because the collision is SILENT and the
wreckage is PLAUSIBLE — a PR description full of real prose about a real PR,
just the wrong one. Nothing in the write path could tell my file from a file
with my name on it, and the natural filename is exactly the one every lane
picks. It scales with parallelism: the more lanes, the likelier two reach for
pr-body.md in the same minute.
What it is. lane-scratch <item-id> mints and prints
<scratchpad>/lanes/<item-id>/, idempotently, with an owner stamp; the Driver
runs it in the same breath as the assignment and names the path in the kickoff.
canon/guard-lane-scratch.py is the mechanical half: a PreToolUse guard
(Write|Edit|NotebookEdit and Bash) that refuses to CREATE a file at a
scratchpad root while a lanes/ directory exists, and refuses to overwrite a
root file a DIFFERENT writer already wrote, naming the other writer. Lanes are
told apart by their own sub-agent transcript; the session id is the same for all
of them. Every permitted write is recorded in <scratchpad>/.writers.jsonl.
Liveness: why signals is EMPTY, and what would fill it
status is proposed because nothing has run it yet, not because nothing
installs it. The first draft of this entry said the guard was inert pending a
hand-added line — that was true of the first draft of the PR and a fresh-context
review called it the blocker: ~/.claude/settings.json is in no repo, so a hook
whose installation is a remembered step is a hook that never runs, while the two
SKILL.md files (which ARE symlinked live) tell every Driver and Worker it will.
loop-doctor now owns both halves — its canon map links
~/.claude/hooks/guard-lane-scratch.py and its check 3 registers the command
under PreToolUse with matcher Write|Edit|NotebookEdit|Bash, re-registering it
after a reset --hard of ~/.claude. It goes active on its first run in a new
session (hook config is snapshotted at session start).
The liveness question here is not "did it tick" — it is "is it wired", and the
catalog cannot ask that today. evaluateSignal supports exactly four types —
file-age, log-match-age, launchd, self (the SystemSignal.type union in
relay/src/systems/catalog.ts)
— and none of them reads a pattern out of a config file. Declaring a fifth
spelling would render unknown signal type, which is a worse entry than an
honest empty one. What would fill it: a file-match type (path + pattern,
ok when the pattern is present), at which point the signal is
~/.claude/settings.json matching guard-lane-scratch and this entry goes
active with no other edit. Until then lane-scratch --check answers it for a
human, and loop-doctor --check answers it for the fleet.
⚠️ A per-write liveness signal would be a trap, which is why none is proposed.
The natural candidate — age of the last blocked collision — reads STALE exactly
when the fleet is healthiest, and the ledger it would read lives at a
session-specific path that no fixed signal can name. An unwired guard and a quiet
one are indistinguishable from the inside; that is the same reason
lane-scratch --check exits 4 (could not measure) on an absent ledger and
never 0: nothing was recorded is not nothing collided.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
review webhook + subscriptions
| What it does | SINCE 2026-10-23: a review verdict posted as the bot's finished ISSUE COMMENT (claude[bot], `Claude finished`, a 🔴/🟡/🟢 / NO HIGH FINDINGS / HIGH marker, or since dotclaude#304 a `## GREEN` / `## RED` / `## YELLOW` heading as the agentflow-relay and local-bin reviewers write; dotclaude#303) wakes subscribers like a formal review, once per (PR, comment id, verdict), head sha read via `gh`, so agent-harness reviews (bodyless formal reviews) now wake; a 🔴 comment and a CHANGES_REQUESTED review stamp as one ruling, so propflowai still wakes once. SINCE 2026-10-24 (dotclaude#305) plan review runs only for its listed repos (PLAN_REVIEW_REPOS: propflowai, agent-harness, factory, exactly the repos it ran for before), so agentflow-relay, local-bin and (since 2026-11-04) tenio carry the same webhook as agent-harness (hooks 702666263, 702666268 and 707500453, same URL, secret and events) for review wakes only; a PR opening there logs `plan_review.skip_not_listed` and starts no plan review. pr-autosub arms any PropFlow-Technologies/<repo> PR (dotclaude#302), not only propflowai; agent-smith/agent-harness fold through review-sub/listener aliases. Receives GitHub review/merge events over a Cloudflare tunnel, debounces them per-PR, then for every subscribed session writes a signal file and types a wake into that session's pane — so a session waiting on a review never polls for it. Since 2026-10-02 (ADR-0039) it also follows every propflowai PR through its whole life: a per-PR status file (~/.claude/review-subs/status/<owner>-<repo>-<pr>.json, stage history + timestamps, read by `review-sub status <pr>`) for every PR, and a coalesced wake to subscribers on checks failed at head, auto-merge armed, entered the merge queue, KICKED OUT of it (with GitHub's reason), and LIVE IN PRODUCTION (deployment_status success on environment Production; PRs = the (#N) squash subjects between the previous and new production SHA, one compare call). Sessions are subscribed automatically by the pr-autosub PostToolUse hook on their own `gh pr create` (`review-sub arm --auto`, 36h TTL). SINCE 2026-10-27 ATTACHMENT IS REQUIRED, NOT OFFERED (Gera, after propflowai #22517/#22518 sat red ~7h with `subscribed: 0 session(s)`: `gh pr create` 502'd, the helper fell back to `gh api .../pulls`, and the hook matched only `pr create`). Three layers: (1) the hook (dotclaude#328) also treats `gh api repos/PropFlow-Technologies/<repo>/pulls` POSTs as creates, looks a 5xx'd create up by head branch (`--author @me`, created <15 min), and reports any create it could not attach to the session as a PostToolUse `decision: block` naming the `review-sub arm` to run (logs `pr_autosub.looked_up` / `pr_autosub.no_pr` / `pr_autosub.failed`); (2) `pr-merge` refuses to arm (exit 9) a gera-propflow PropFlow-Technologies PR that `review-sub attached` (0 = live sub, 1 = none, 4 = unknown; asks the listener host from the MacBook) does not confirm; (3) pr-autoqueue's `pr-reconcile` sweep re-attaches any orphan on every pr-autoqueue tick via `review-sub arm --owner <full sid>` (armer kind `reconcile`; a sub whose owner session has ended does not count), and `pr-stuck-wake` wakes the owner of a PR stuck 30 min at one head. Both sidecars' signals live on the pr-autoqueue row. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | ~/.claude/services/review-webhook/listener.js, subscriptions in ~/.claude/review-subs/ (status files in its status/ subdir, last production SHA in status/_production.json), armed by ~/.local/bin/review-sub and by the PostToolUse hook ~/.claude/hooks/pr-autosub.sh (log ~/.claude/jobs/pr-autosub/autosub.log) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → review webhook + subscriptions |
Liveness signals
| Signal | How it is read |
|---|---|
| listener running | launchd · com.propflow.review-webhook |
| tunnel running | launchd · com.propflow.cloudflared-claude |
| events arriving | log-match-age · ~/.claude/services/review-webhook/listener.log · matches event.in · stale after 360 min |
| PR stages being recorded | log-match-age · ~/.claude/services/review-webhook/listener.log · matches pr_stage.recorded · stale after 360 min |
| production deploys reaching PRs | log-match-age · ~/.claude/services/review-webhook/listener.log · matches prod_deploy.live · stale after 360 min |
| owner notices being delivered (never typed, since ADR-0076 slice 4b) | log-match-age · ~/.claude/services/review-webhook/listener.log · matches notice.delivered · stale after 360 min |
| PR openers being subscribed | log-match-age · ~/.claude/jobs/pr-autosub/autosub.log · matches pr_autosub.armed · stale after 360 min |
| stuck-PR sweep running | log-match-age · ~/.claude/services/review-webhook/listener.log · matches pr_stage.sweep · stale after 30 min |
| question guard checking answers | log-match-age · ~/.claude/jobs/ask-wake-guard.log · matches post ok · stale after 4320 min |
Why this was not in the catalog until 2026-09-12. It predates the catalog
and nobody added it, which is exactly the failure the registry exists to catch:
review-turns forbids polling and points every session at this system, so it
has been load-bearing for the whole fleet while being, by this file's own rule,
a system that does not exist. It is listed now because review-sub was built
against it.
It also triggers ci-factory-mq (2026-10-01). On a propflowai pull_request
enqueued, dequeued or closed+merged event, triggerCiFactoryMq spawns
~/.local/bin/ci-factory-mq detached (stdio ignored, unref'd, every step in a
try/catch that only logs), throttled so a burst produces one run 10 s after its
first event. Since 2026-10-02 a check_suite / workflow_run completed whose
head_branch is main also arms it (triggerCiFactoryMqOnMain, at most once per
CI_FACTORY_MQ_CI_GAP_MS = 60 s, called before the not-a-verdict skip and never
awaited), so main's test progress on /a/ci-factory stays current between merges. Log lines: ci_factory_mq.fire, .skip_missing, .spawn_error. A
missing or crashing script cannot reach the review path; that was proven with an
isolated old-vs-new differential run before the restart.
Every stage of a PR, since 2026-10-02 (ADR-0039).
The listener records each propflowai PR's stages into a per-PR status file and
wakes subscribers on checks failed at head, auto-merge armed, queued, kicked out
of the queue (with GitHub's reason) and live in production, coalesced per PR over
30 s. The repo webhook gained deployment_status for the last one (added with
add_events; URL, secret and the other events untouched). Sessions subscribe by
opening the PR: the pr-autosub PostToolUse hook runs review-sub arm --auto
on a gh pr create that prints a propflowai URL. review-sub status <pr> reads
the file. Log lines: pr_stage.recorded / .wake / .no_subs / .error,
prod_deploy.live / .baseline / .skip_not_ahead / .compare_error. Gap,
stated in the ADR: a PR opened outside a Claude gh pr create gets the status
file but wakes nobody unless someone arms it.
⚠️ deployedAt is edited IN PLACE. The dotclaude copy was synced FROM it on
2026-10-02 (gera-propflow/dotclaude#145) and drifts again with the next in-place
edit; sync live -> repo, never the reverse. On
2026-10-01 the live listener.js was 552 lines added and 344 removed ahead of
services/review-webhook/listener.js in gera-propflow/dotclaude, at both HEAD and
origin/main. The repo's last commit to the file is from mid-September. Back up to
listener.js.bak-<epoch> beside it, edit, node --check, then
launchctl kickstart -k gui/$(id -u)/com.propflow.review-webhook. Restoring the file
from dotclaude would silently drop ~900 lines of live behaviour.
The liveness signal is event.in, not a wake. Deliberate. Wakes are rare —
they fire only when a subscribed session's PR gets a verdict — so keying
liveness on one would read down through every quiet hour and up only by
luck. event.in is logged for every delivery GitHub makes, including the ones
the listener then skips, which is the question this row actually asks: is the
tunnel still terminating and is the process still reading it. The 6h window is
generous on purpose; a fleet that opens no PRs overnight is not a broken tunnel.
⚠️ A GREEN ROW HERE DOES NOT MEAN A WAITING SESSION WILL BE WOKEN, and that
gap is the reason review-sub exists. The listener can be perfectly healthy
while every subscription on disk is undeliverable — measured on 2026-09-12
across 98 live subscriptions: 56 with no wake target at all, 42 with a wake
target and no record of who armed them, none routable. The cause was that a
Driver's lane is an Agent-tool sub-agent with no pane of its own, so the arming
block resolved its parent's pane and the wake landed on the Driver while the
lane waited for a signal it could not receive.
The catalog cannot ask that question today, and an entry that implied it could
would be worse than one that says so. evaluateSignal supports file-age,
log-match-age, launchd and self — none reads a JSON field out of a
directory of files. What would fill it: the same file-match type the
lane-scratch entry above asks for, or a dir-predicate signal, at which point
this row could assert every live subscription names an armer. Until then the
answer is a command, not a row:
review-sub list # every subscription, each diagnosed OK / UNATTRIBUTED / DEAF
DEAF is a subscription with no wake target: nothing types anywhere, and only a
session still polling for the passive signal file would ever notice. DEAF and
OK are told apart on purpose rather than both counted as "subscribed", for the
same reason lane-scratch --check exits 4 rather than 0 on an absent ledger —
nothing will be delivered must not render the same as nothing is wrong.
SINCE 2026-10-29 the typed wake is GONE (gera-propflow/dotclaude#350, ADR-0076 slice 4b): wakeSession hands one coalesced NOTICE per owner session per 30 s, across all its PRs, to local-bin deliver (quiet → PAUSED → attended notice file → messaging socket); wake_session.sent no longer exists in the log, notice.delivered does (signal above). The paragraph that follows is the history.
2026-10-29, earlier the same day (ADR-0076):
Gera's ruling after 110 wakes were typed into his main pane in one night — a
wake for a worker goes straight to that worker (its own background wait on the
signal file; the harness resumes it), nothing is ever typed over a human, and
quiet (next entry) holds everything. Slice 1 (dotclaude#346): wakeSession
checks the quiet markers FIRST, before PAUSED and before any tmux, and logs
quiet.held instead of wake_session.sent. The slices that follow replace the
typed wake with a coalesced owner NOTICE (one per owner per 30 s across PRs; a
deploy is one line) delivered as additionalContext on the session's next
action, and delete buildTmuxWakeArgs, the dialog gate and the wake cap; the
wake_session.sent reading above describes the path until then.
Appendix
- https://github.com/PropFlow-Technologies/propflowai#readme — the README (no docs declared in SYSTEMS.md yet; add some)
quiet (hold every machine wake into a session; one digest on lift)
| What it does | LANDING 2026-10-29 (slice 1 of ADR-0076): this entry reads healthy only once gera-propflow/dotclaude#346 (the listener hold) is merged AND deployed into the live ~/.claude and the listener restarted, and PropFlow-Technologies/local-bin#704 (the CLI) is merged; until then the launchd signal below says only that the process that WILL honour the marker is up. A fact on disk, distinct from PAUSED: PAUSED means the model stands down (driver-pause); quiet means humans talk and machines do not, so the human converses normally while nothing is typed into or injected at the session. Markers under ~/.claude/jobs/quiet/: `ALL` (the fleet switch; it does NOT pause Drivers), `<sid>` (one session, full lowercase uuid; JSON by/at/reason/pane) and `pane/<tmux>` (the pane that session's wakes would be typed into — the belt for a subscription whose sid does not resolve). Set by local-bin `quiet on|off|status [--sid S | --all]` (local-bin#704; a caller off the listener's host runs it there over ssh), and, once dotclaude's notices hook lands (ADR-0076 slice 2, branch `notices-inbox-hook`; grep `slice 2` when it does), by `/quiet`, `/quiet off` and the bare word `quiet` / `resume` typed as a whole prompt. Honoured by the review-webhook listener first (dotclaude#346: checked before PAUSED and before any tmux, local and remote alike, because the markers live on the listener's box), and by every other injector as each moves onto `deliver()`. A held event is one JSON line of `~/.claude/jobs/quiet/<key>.inbox.jsonl` (at, repo, pr, kind, verdict, sha, stages, text; key = sid, else `pane-<tmux>`); the listener logs `quiet.held`, and a hold it cannot append fails CLOSED (`quiet.hold_failed`, nothing typed; the stage is still in the status file). Workers are untouched: their signal files are still written and they still resume. `quiet off` removes the marker FIRST, renames the inbox aside, then coalesces it into ONE digest — a clause per deploy, per kind, per verdict, never per event — printed and written to ~/.claude/jobs/notices/pending/<sid>.jsonl, the shape wt-merged-inbox delivers as additionalContext on the session's next action. `off --all` lifts every marker and digests every inbox to its own session. Decision record: agentflow ADR-0076. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/local-bin |
| Deployed at | ~/.local/bin/quiet (markers and inboxes in ~/.claude/jobs/quiet/); the hold in ~/.claude/services/review-webhook/listener.js (quietHold / holdForQuiet) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → quiet (hold every machine wake into a session; one digest on lift) |
Liveness signals
| Signal | How it is read |
|---|---|
| the holder (review-webhook listener) running | launchd · com.propflow.review-webhook |
Why one signal, and why it is the listener's. Quiet is a marker and a CLI;
nothing of its own runs. What proves it works is that the process that would
otherwise type — the listener — is up and reads the marker on its next fire.
A per-event signal (quiet.held recent) would read stale exactly when the
fleet is healthy and nobody is quiet, which ADR-0009 forbids; the proof that
the hold itself works is the test each side's PR carries (dotclaude#346 for the
hold, local-bin#704 for the CLI and the digest), run by the PRs that change
either side.
Appendix — everything linked from this entry
gh-app-token (installation tokens for the propflow-agent-smith GitHub App)
| What it does | Prints a one-hour installation token for the GitHub App propflow-agent-smith (installation on PropFlow-Technologies, all repos; actions/checks/contents/pull_requests/issues/deployments/administration read, statuses write) so the mini's scheduled GitHub readers spend the App's 15,000 REST / 10,000 GraphQL an hour instead of the one shared user account's 5,000, which hit its limit at 2026-10-06 03:31Z and starved every agent. RS256 JWT signed by openssl over the key in ~/.config/github-apps/propflow-agent-smith/ (600 in a 700 dir), exchanged at /app/installations/<id>/access_tokens; cached 600 in ~/.cache/gh-app-token/, re-used until under 10 minutes remain, one mint under an fcntl lock; exit 1 and one stderr line on any failure; nothing prints or logs the key, JWT or token except the token to stdout. Consumers go through lib/gh_auth.py (Identity.run; per-call scope since #437: a READ naming only PropFlow-Technologies owners runs as the App, anything else as the user): a failed mint or a 401/403-by-integration on the token falls back to the user's gh auth, logged once per run, and the token is forgotten (--forget); a 403 rate limit never demotes. Readers on it: ci-factory-mq / -health / -flow (#431, #435), puller, worker-cleaner (`cleaner`), cerebrus-gate, merge-evidence-gate (#437), propflow-docs bin/refresh-tracker (#315, #316), Agent Smith's hot gh helpers (agent-smith #821, its own agent_smith/gh_auth.py). The interim ~/.config/github-apps/propflow-agent-smith/token.sh execs this. What stays on the user account on purpose: repos outside PropFlow-Technologies (the App sees only the org it is installed on: a personal repo answers HTTP 404 with its token, so readers scope by owner and run a call as the App only when every owner it names is in the org), writes other than statuses, org billing, org-admin and user-scoped calls (`gh api user`, `--author @me`). |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/gh-app-token and ~/.local/bin/lib/gh_auth.py; cache ~/.cache/gh-app-token/; log ~/.claude/services/gh-app-token/mint.log |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → gh-app-token (installation tokens for the propflow-agent-smith GitHub App) |
Liveness signals
| Signal | How it is read |
|---|---|
| a token was minted recently (one mint an hour is healthy) | log-match-age · ~/.claude/services/gh-app-token/mint.log · matches mint ok app=propflow-agent-smith · stale after 180 min |
Why an App and not a second user token. A token on another user account is the same
5,000/hour shape one more time, tied to a person's login, and it starves the same way. An
installation token is the organisation's, has three times the REST budget and twice the
GraphQL budget, and cannot do the things the collectors must never do by accident (no
gh api user, no merges, no org billing: HTTP 403 "Resource not accessible by integration").
The cutover order was by load: ci-factory-mq first (a floor pass every minute, the heaviest
reader), then its health and flow passes, then the rest of the scheduled readers.
Appendix — everything linked from this entry
ci-factory-mq (GitHub merge queue + its pass rate -> R2 for /a/ci-factory)
| What it does | One GraphQL call (mergeQueue on propflowai main: position, state, enqueuedAt, PR number/title/author) written as {generatedAt, totalCount, entries} to R2 propflow-podcasts/ci-factory/mq.json, then read back to prove it landed. propflow-docs functions/api/factory.js overlays it on the CI Factory snapshot when it is under 20 minutes old: every queued PR gets the merge-queue station, counts.mq and counts.approved are set. Stale or missing serves the snapshot unchanged. Triggered by review-webhook on enqueue/dequeue/merge plus a 15-minute launchd reconcile; runs never overlap (flock plus a rerun flag, so the last event of a burst is not lost). Safe to run by hand. When GitHub has no merge queue on main it writes enabled:false and total 0 rather than erroring. The reconcile runs it with --health, which then runs the sibling ~/.local/bin/ci-factory-mq-health (own lock): 7 days of CI merge_group runs, each new red run's failing-job logs read once (cached per run id, at most 60 per run), a why-it-went-red bucket per run, failures on their own vs inherited from the run ahead, passed-on-retry vs needed-a-fix, written to R2 ci-factory/mq-health.json (/api/factory passes it through while under 45 min old). Since 2026-10-02 that JSON also carries `culprit` {pr, removedAt, runUrl, tests, reasons}: the PR GitHub last removed from the queue for failing checks, confirmed against the PR timeline (a failed_checks RemovedFromMergeQueueEvent at that run's own queue commit, never inferred from red runs), at most 8 timeline calls per pass; the CI Factory page shows it as one line, Last queue culprit: #X. Since 2026-10-02 every run also does a FLOOR PASS (at most once a minute, its own try/except and log line, after the queue write): R2 ci-factory/floor.json with `top` (the in-progress heavy-lane run, post-merge-heavy.yml at :07/:37, the one full test of main since push:main stopped running the full suite in #15194 and the merge queue runs the full set before merge; phase testing/failed, non-skipped jobs done/total, typical minutes = median of the newest 6 green heavy runs; CI_FACTORY_BUS=push puts main's CI push run on top instead), `aboard`/`boarding` (PRs merged after the ci/heavy-green tag up to that run's head SHA / merged after it, or everything since the tag when no run is in progress, from main's commit messages; the tag is read in the same history GraphQL query), `other` (the other kind of run if in progress) and `approved` (one plain reason per approved-not-queued PR: hold, conflict, required check failed/running, review pending on a new head, review 🟡 = no approval, dismissed approval, auto-merge not armed, blocked, entering). /api/factory's mergeFloor overlays it while under 20 min old: the page's On main station is two lanes (full run now / boarding next; counts cover all of main, dots are the PRs still on the station since deploys do not wait for the full run; the overlay moves no PR between stations) and Approved dots say why on hover. Cost per floor pass: 3 REST + 1-4 GraphQL (one batched status query, then the check status of the PRs that need it, 10 PRs per call, Approved capped at 30; since 2026-10-03, when one-query-per-required-name hit GitHub's per-query resource limit for a day). Since 2026-10-03 (ADR-0041) the 6-hourly flow pass (sibling ~/.local/bin/ci-factory-flow) also keeps HISTORY: R2 ci-factory/history.json, one object 60 days deep — the flow readings (a silence over 9 h between two is written as an explicit no_reading row, so a dead collector is a gap, never a clean zero), one record per UTC day of main's full test (post-merge-heavy.yml, event=schedule: runs, red hours, red spells from the first failed run's start to the next green run's completion, time-to-green; cancelled is no verdict; a day with no completed run is no_reading; 2 REST calls a pass) and the real reds of the last 72 h (one per culprit, one per infra run, facts only, ejection null until the health pass confirms it). /api/factory passes a 14-day view through as snap.ciHistory (never snap.history: the CI Factory collector's own factory.json already owns that key, the station-count series the full page's trends box draws; propflow-docs #297) for the scorecard strip on /a/ci-factory-floor (the five measures of docs.propflowai.co/a/ci-quality-grade-2026-10 with their targets and trend) and the reds chart. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/ci-factory-mq and ~/.local/bin/ci-factory-mq-health, LaunchAgent ~/Library/LaunchAgents/co.propflow.ci-factory-mq.plist, state + log in ~/.claude/services/ci-factory-mq/ |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → ci-factory-mq (GitHub merge queue + its pass rate -> R2 for /a/ci-factory) |
Liveness signals
| Signal | How it is read |
|---|---|
| reconcile scheduled | launchd · co.propflow.ci-factory-mq |
| the queue reached R2 recently | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches ok total= · stale after 30 min |
| queue health reached R2 recently | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches health ok · stale after 45 min |
| main's test bus reached R2 recently | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches floor ok · stale after 30 min |
| merge queue flow numbers reached R2 recently | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches flow ok · stale after 780 min |
| the CI history (flow readings, main's red spells, the reds) reached R2 recently | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches history ok · stale after 780 min |
| the floor pass reads GitHub as the propflow-agent-smith App (not the shared user account) | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches floor ok auth=app · stale after 30 min |
| the health pass reads as the App | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches health ok auth=app · stale after 45 min |
| the flow pass reads as the App | log-match-age · ~/.claude/services/ci-factory-mq/run.log · matches flow ok auth=app · stale after 780 min |
Successor (2026-10-06, ADR-0044). factory.propflowai.co replaces /a/ci-factory and
/a/ci-factory-floor with one D1 event store filled by factory-collect; these three collectors
keep writing R2 until each page is covered and is retired by its own PR, with a same-window
totals control. See the factory-collect entry below.
Why a 15-minute reconcile on top of the webhook. A queue position change or a
QUEUED -> AWAITING_CHECKS change sends no webhook, and a quiet spell (main red, so
nothing enqueues) would leave mq.json past the 20-minute freshness gate even though
it is still right. That is 4 GraphQL calls an hour, well inside budget.
It exists because the CI Factory's own collector is not on this machine. Its
snapshot (ci-factory/factory.json) is written by a collector whose header names
~/.claude/ci-factory/collect.py / co.propflow.ci-factory. Neither exists on Gera's
mini or MacBook, and every commit to the page is Fede's, so it most likely runs on
Fede's machine. On 2026-10-01 that snapshot labelled 22 of the 50 queued PRs as
queued and filed the rest under Needs a fix and Build & test. This entry is the
overlay, not a second collector: it writes the queue and nothing else.
There is no automatic re-queue of "innocent" PRs, on purpose (2026-10-02). It was
proposed after a PR broke two Clara guard tests in the queue and the PRs behind it
looked collateral. The replay said no. Of the 66 failed_checks removals from the
queue's start (2026-09-30 22:40Z) to 2026-10-02 14:30Z, all 62 that can be judged were
the EARLIEST failing entry in their window: every run ahead of them had passed. The
other 4 cannot be judged. They were all on day one (last 2026-10-01 19:10Z), removed
while their own CI run was still going, and that run later went green, so whatever
check failed them is not in the CI workflow this chain analysis reads. Of 1,044 red runs that had a red
run ahead of them, GitHub removed none; it rebuilt them, as designed. The PR cited as
"innocent, dropped twice" (#15340) was the culprit all three times. Its own diff
failed a loading-shell drift test, then type check twice, against a moved main, while
the identical base without it passed. The proposed rule "the failing tests pass on main
and the PR's diff does not touch them" would have re-queued it, because the drift test
scans the repo and is not in the diff. Revisit if a removal ever shows a failing run
ahead of it, or if the day-one kind (removed with its own CI still green) comes back.
Amended 2026-10-04 (rulings fb239c15e -> f3eed9afe; propflowai #16344 + #16423): ONE
re-entry per commit after a runner kill, and nothing re-queues it but the PR's own arm.
The no-auto-requeue rule above is about INNOCENCE, which a replay of the queue could not
establish; this exception does not claim it. propflowai's
.github/workflows/queue-reentry-guard.yml asks scripts/ci/queue-reentry-infra-kill.sh
(#16344, the one owner of the classifier) whether a failed_checks removal was a runner
kill: a job whose log carries The runner has received a shutdown signal, every other
failure being an aggregate that restates it. A build out-of-memory kill carries no such
line and stays a real failure. On a kill the guard leaves the auto-merge arm ALONE (it
never arms and never enqueues): a standing arm re-adds the PR through GitHub and the sole
armer's gates, and a PR nobody left armed stays out until someone arms it. #16423 bounds
it: the kill is honoured only as the head's FIRST failed_checks removal (counted from
GitHub's own timeline, so it needs no store and a new push starts at zero), and at most
ONE add after that removal is let in. A second kill at the same commit is disarmed like
any failure, so a head that keeps losing machines cannot loop the queue. A removal outside
that budget is disarmed before the log read, and a re-add that slipped in while a real
failure was being read is dequeued: at most one extra queue run. Origin: #16207 at
41d62c11, removed when queue run 37160791435 lost its CI Checks machine. Signals: none
added, and this entry's own are unchanged. The guard is a GitHub workflow with no
artifact on this machine, so no signal type in this catalog can watch it. It is visible per
event as its run annotations and the PR's <!-- queue-reentry-guard --> comment. The
signals above watch the overlay this entry publishes, which the guard does not touch.
Appendix
- https://github.com/gera-propflow/local-bin#readme — the README (no docs declared in SYSTEMS.md yet; add some)
factory-log (one line into factory.propflowai.co/log, from the shell)
| What it does | `factory-log "<plain-English one-liner>" --measure queue|main|live|cost [--pr N] [--who <name>]` POSTs one effort to factory.propflowai.co/api/efforts through the Cloudflare Access service token in ~/.config/factory/access-service-token.env (read into the headers, never printed); the API verifies Access's JWT and records the author as service:<client id>, --who is only the name shown beside the line. Prints the created id; every failure is one stderr line and a distinct exit code (2 usage and nothing sent, 3 no token, 4 the API refused with its reason, 5 network); --dry-run prints the request and sends nothing. The Log measures each line's effect 48 h after its PR merges, from the same tables Health reads. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/factory-log; log ~/.claude/services/factory/factory-log.log; invoked by hand or by a lane, no schedule |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → factory-log (one line into factory.propflowai.co/log, from the shell) |
Liveness signals
| Signal | How it is read |
|---|---|
| a line reached the Log this week | log-match-age · ~/.claude/services/factory/factory-log.log · matches sent id= · stale after 10080 min |
The shell half of Gera's effort Log. The Log's screen and the + Log an effort drawer
(PropFlow-Technologies/factory #21, #26) write the same line through a Google login; this tool is
how a lane, Agent Smith or a cron writes one without a browser, which is what "who's doing what"
needs to stay true. It owns nothing on the server: factory#17 is the table and the routes,
factory#35 the identity rule (verified Cf-Access-Jwt-Assertion, never the plain email header).
What it deliberately does not do: read the Log (that is the page), retry (a failure is one line
and an exit code, the caller decides), or measure anything (the store does, 48 h after the line's
PR merges).
Appendix — everything linked from this entry
factory-collect (the CI factory collector on the mini -> D1 behind factory.propflowai.co)
| What it does | Every 15 minutes runs PropFlow-Technologies/factory's collector (collector/factory_collect.py) against the D1 event store behind factory.propflowai.co, as the propflow-agent-smith GitHub App: a `sweep` every tick (GitHub reads: runs, PR timelines, deployments, main commits; plus the interim spend source, Fede's factory.json read from R2) and a `classify --hours 30` pass every other tick (gated to once per 20 minutes, skipped when the App has under 2,500 REST left this hour) that runs propflowai's own scripts/ci-flow/flow-baseline.ts over a rolling 30-hour window and imports its per-run classes. Each command writes one `collections` row, ok or not, so a missed collection is a row and never a zero. The shim owns only the schedule, the fcntl lock, the two dedicated runtime worktrees (~/code/PropFlow/.factory-runtime of the factory repo and ~/code/PropFlow/.factory-flow-runtime of propflowai, each forced to origin/main and refused unless clean, the ci-factory-flow pattern), the GitHub identity (lib/gh_auth.py, GH_TOKEN handed to the collector) and the log line. Inert until ~/.config/factory/d1-id holds the database uuid (one skip line, exit 0). |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/factory-collect, LaunchAgent co.propflow.factory-collect (launchd/co.propflow.factory-collect.plist in local-bin, 900 s), state + log in ~/.claude/services/factory/ (run.log, run.lock, classify.last-ok, flow-cache/); runtime worktrees ~/code/PropFlow/.factory-runtime and ~/code/PropFlow/.factory-flow-runtime |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → factory-collect (the CI factory collector on the mini -> D1 behind factory.propflowai.co) |
Liveness signals
| Signal | How it is read |
|---|---|
| collector scheduled | launchd · co.propflow.factory-collect |
| the GitHub sweep reached D1 recently, as the App | log-match-age · ~/.claude/services/factory/run.log · matches sweep ok auth=app · stale after 30 min |
| the classify pass (flow-baseline's run classes) reached D1 recently, as the App | log-match-age · ~/.claude/services/factory/run.log · matches classify ok auth=app · stale after 45 min |
What it writes, and what it does not own. The D1 event store behind
factory.propflowai.co (PropFlow-Technologies/factory: runs, run_class, jobs,
failures, prs + pr_events, main_commits, deployments, cost_days, collections),
events never totals; the app's functions compute every number a page shows from those rows
at read time. The run classes (productive, real_failure, inherited_from_main, cascade_red,
discarded, infra, ...) stay owned by propflowai scripts/ci-flow/flow-baseline.ts: the
classify pass runs that script and records what it said, nothing here re-implements a
class. The why-it-went-red buckets stay owned by ci-factory-mq-health (the collector's
import-health imports the installed module's bucket_of; the rule is never copied). The
sibling ci-factory-mq collectors keep writing R2 for /a/ci-factory and /a/ci-factory-floor
until factory.propflowai.co covers each page (ADR-0044).
Runs carry an app (factory#125, 2026-11-02). propflowai is one repo with several apps
(propflowai ADR-0153), so every runs row carries app, decided by the workflow file alone:
ci.yml + post-merge-heavy.yml = product, ci-site.yml = site (Website),
ci-internal.yml = internal (Dev tools). The rule is the factory's
functions/_lib/apps.ts; the webhook and this collector stamp it at ingest, migration 0015
backfilled the older rows, and a control pins the copies together. The sweep now also lists the
two app workflows (pull_request + push) and reads their job lists, because their production
deploy is a job inside the same run (Product's deploys stay GitHub's Production deployments).
The Health tab's App picker (Product · Website · Dev tools) reads it.
Appendix — everything linked from this entry
plate, formerly matrix (plate.propflowai.co — each person's week as Tetris blocks, written from the `plate` CLI)
| What it does | One column per person (their whole week), one block per thing on their plate sized by the share of the week it still takes, with a days-left countdown; free space on top, blocks settle at the bottom. Written from the shell — `plate add "<≤4 words>" 40% 5d` (`matrix` still works as an alias that notes the rename on stderr), `small`, `half`, `done [N%]`, `days`, `size`, `rename`, `give`, `--as <person>` — through the Cloudflare Access service token in ~/.config/plate/access-service-token.env; the page only reads. Pages Functions over D1 `matrix` (one `items` table; the board is derived from open rows, `due` is a date, `done_pct` shrinks the block); writes are signed by the VERIFIED Access JWT and a service token must name the board; at most 5 big blocks per person (a 6th is refused naming the 5); a block untouched for 24 h fades. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/plate |
| Deployed at | Cloudflare Pages propflow-matrix → plate.propflowai.co, plus the old matrix.propflowai.co (browsers are sent to plate; its /api/* is served as-is for un-pulled CLIs) (Access app Matrix covers both hosts: Google, propflowai.co + service token matrix-cli); D1 matrix 29983433-…; CLI ~/.local/bin/plate → ~/code/PropFlow/plate/cli/plate (alias ~/.local/bin/matrix → plate/cli/matrix); deploy on push to main. The Pages project, D1 database, Access app and token keep their matrix names (infra; neither a Pages project nor a D1 database renames in place) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → plate, formerly matrix (plate.propflowai.co — each person's week as Tetris blocks, written from the `plate` CLI) |
Liveness signals
| Signal | How it is read |
|---|---|
| someone moved a block in the last 3 days | log-match-age · ~/.config/plate/writes.log · matches sent id= · stale after 4320 min |
Gera's "Tetris board" (Oct 1 and Oct 5 calls: a weekly view of 2–3 priorities per person, so everyone can
see who is consumed by what). Named Plate on 2026-10-15 ("it's on my plate"); it was Matrix, for the Tetris playfield. Same shape as factory
(Pages + Functions + D1 behind Access, a stdlib CLI with service-token headers and a named User-Agent).
Sessions never write to it on their own. matrix-watcher, which moved a block a person LINKED to a Driver
(matrix link), was retired 2026-10-10 (see below), so nothing moves a linked block any more.
Appendix — everything linked from this entry
matrix-watcher — RETIRED 2026-10-10
Retired (Gera 2026-10-10, fleet cleanup step 2). In its whole life (556 passes) every pass logged
linked=0 moved=0: no Matrix block was ever linked to a Driver. Its 6 h launchd
job was never installed; it ran only as a detached spawn from updater after
each success, which is now a no-op (shutil.which finds nothing). Script, plist,
test and README section deleted in local-bin; ADR-0047 carries the amendment.
Rebuild it if someone actually links a block.
Cost check + $100/day spend floor
| What it does | Every 3h, reads GROSS daily usage per service (GitHub billing usage today + yesterday, plus the AWS Cost Explorer delta/anomaly check) and posts a SPEND ALARM with the alarm GIF to #spending (C0C50G2EBMK) when any single service passes $100 in a day, at most once per service per day. The AWS anomaly half NO LONGER posts a verdict of its own on every run (agent-smith#957, merged 2026-10-14T03:28Z as 6d4d9299; Fede ruled 2026-10-12 for major things only, budget-relative): findings ride the budget alarm as the up-to-DRIVER_LINES '↳' lines, or go out as ONE batched message when an unseen finding is worth SOLO_ALERT_SHARE_OF_DAILY_BUDGET (0.25) of the day budget on its own, and on a quiet AWS day NOTHING posts — '_post_aws_alarms' calls that branch 'the run row, not Slack, is the heartbeat'. Detection is unchanged; only where it SPEAKS moved. Still posting on every occurrence: each '<what> FAILED — <what was lost>' could-not-look page, and the full legacy verdict when the account budget cannot be read (unknown never demotes). All of it through one constant, cost_anomaly.SPENDING_CHANNEL. #alerts carries system alerts only (Gera, 2026-10-05; agent-smith#798) |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.cost_anomaly + spend_github in the Smith worker; Temporal Schedule cost-anomaly-daily (propflow-tools-prod, scripts/register_cost_anomaly_schedule.py); dedupe ledger ~/.claude/smith-state/cost-alarm-posted.json |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Cost check + $100/day spend floor |
Liveness signals
| Signal | How it is read |
|---|---|
| worker running | launchd · co.propflow.smith-worker |
| check completing | log-match-age · ~/Library/Logs/smith-worker.err.log · matches cost_anomaly done · stale after 240 min |
Why it exists. On 2026-09-28, GitHub Actions cost $995 in one day ($1,875 billed
in four days) and nobody was told. The only cost check read AWS alone, and its
schedule failed every run: the IAM user gera-dev lacks ce:GetAnomalies. Gera's
rule is absolute: any one service over $100 in a day pages. The check reads
gross usage. GitHub's net bill was $0 for 24 days while included minutes
covered a gross that grew fourfold, and propflowai's spend-snapshot cron records
that billed figure, so an alarm on that feed would have stayed quiet. New vendors
plug in as one small SpendProvider file each.
Reading the signals, and the ledger-freshness one is GONE. "check completing"
matches cost_anomaly done, logged by activities/cost_anomaly.py immediately
above run.ok(...), after the raise that EXIT_INCOMPLETE takes — so it
covers every run that COMPLETES, not every run that starts: EXIT_INCOMPLETE
raises above it, and the activity's own comment enumerates what takes that branch
(the AWS read, a spend provider, a vendor's billing cycle or budget, a day series,
or any of their posts). So it stays red while any part of the run fails, the
ce:GetAnomalies denial named under "Why it exists" included — though that denial
read as historical, not current, when this was written (2026-10-13).
14400s is 4 h against a 3 h cadence: one missed fire reddens it. It is now the
only liveness signal here, and until 2026-10-13 it could not carry that — the
evaluator tailed a FIXED 64 KB of a log the whole worker writes to, so the marker
left the window minutes after each 3-hourly fire and this row read down nearly
all the time while the check was fine. The window now widens to the signal's own
staleAfterS, fixed in the same PR that retired the ledger signal
(agentflow-relay#343, evalLogMatchAge); _retired_signal_note says why the
ledger signal had to go rather than be re-pointed or widened.
The job manifest points HERE for "did it stop?", so that answer is
load-bearing on agentflow-relay#343 — which is merged and NOT deployed.
propflowai's config/automations.toml entry cost-anomaly-daily names
check completing as the surface that answers whether the schedule stopped,
and post-agent-smith#957 it is the only one in reach: a clean run posts
nothing, and cost_anomaly's own docstring says of the run-row heartbeat that
"nothing in-repo watches for a missing one". That answer is only as good as the
evaluator's window, and #343's repair is a window DERIVED from staleAfterS
and capped at LOG_TAIL_MAX_BYTES (32 MB) — so the margin is
RATE-DEPENDENT, not unconditional: it holds while smith-worker.err.log grows
slower than ~8 MB/h (32 MB over the 4 h budget). Measured 2026-10-13: the
marker sat 2.3 MB back from EOF at 154 minutes old, i.e. ~0.9 MB/h, roughly a
9× margin — and a down that stopped at the cap says so in its
detail, so the day that margin goes is diagnosable rather than silent.
⚠️ #343 merged as 962c3554; the relay's DEPLOYED checkout is
behind main, so until it is deployed this row still renders from the old
fixed 64 KB tail and reads down whatever this file says. Live /api/systems
on the mini, 2026-10-13: check completing → no "cost_anomaly done" in recent log, with the retired a post recorded today still in the payload.
The 4 h budget is tight against a DEPLOY, and that is a threshold question
rather than a logging one. staleAfterS 14400 against a 10800 s cadence is
a one-hour margin, so one missed fire reddens the row. Measured over every
cost_anomaly done line in the log, 2026-09-29 → 2026-10-13: 115
completions, median gap exactly 180 min, and 5 gaps of ~6 h (4.4% of 114
intervals) — each one a Smith worker restart that straddled a scheduled
fire — 5 of 5, each drain→boot window containing the fire instant:
11 m 02 s, 12 m 52 s, 49 m 42 s, 56 m 28 s and 2 h 12 m 42 s. Note the
asymmetry, which is the actual mechanism: the outage only has to straddle the
fire INSTANT, so an 11-minute deploy buys a ~2-hour red (from 4 h after the
last completion until the next one, 6 h after it). GRACEFUL_DRAIN_SEC is
7800 s (2 h 10 m) and the longest window measured here sat right at it, but it
is a CAP on in-flight activities rather than a routine cost — the drain
line itself says "an idle worker exits at once". So the number is not
obviously wrong; it is pointed at a quantity a deploy moves, and widening it
buys blindness to a check that really stopped.
Left as its own row deliberately: changing it changes what the signal MEANS
(ADR-0009's amendment territory), which is not a prose fix.
What nothing here proves, stated rather than left to be assumed. No signal
on this row proves an alarm FIRED or that Slack received anything. Post-agent-smith#957
there is no daily artifact to prove delivery with: a quiet day writes no Slack
post and no ledger key, and the check's own receipt is the Temporal run row,
which lives in DynamoDB and no signal type here can read. So the honest claim
is "the check is completing on schedule", and the delivery path is proved by
#spending having received the last alarm, not by this row.
Appendix
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README (no docs declared in SYSTEMS.md yet; add some)
Daily spending table
| What it does | Every morning, posts ONE table to #spending of what each paid service cost yesterday and month to date, split into usage (gross), credits and what we actually pay, with a 🔴/🟢/⚪ status per row |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | agent_smith.spend_digest (+ spend_counts, spend_gcp) in the Smith worker; SmithSpendingDigestWorkflow, an awaited child step of MorningQueueWorkflow (07:00 America/Chicago, propflow-batch-queue); manifest id spending-digest-daily (alerting=false) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Daily spending table |
Liveness signals
| Signal | How it is read |
|---|---|
| worker running | launchd · co.propflow.smith-worker |
| table posted to #spending | log-match-age · ~/Library/Logs/smith-worker.err.log · matches spend_digest posted ts= · stale after 1560 min |
Why it exists. The $100/day alarm (spend-alarm) only speaks when one service runs
away. Gera asked on 2026-09-29 for the calm daily view as well: one table every morning
with every paid service in it. Usage alone overstates the bill. GitHub ran about $10.5k
gross in September against about $1.9k billed, and AWS usage is fully covered by credits.
So each row shows usage, credits and paid separately, and the header leads with paid.
It reads nothing new. Metered rows come from the alarm's own readers
(cost_anomaly.spend_providers(): GitHub, Vercel, Twilio, Anthropic) and its Cost
Explorer usage read. The other sources are ElevenLabs (grant credits, $0 paid), Temporal
Cloud and Browserbase (estimates from usage × list price, marked est.), and Google Cloud
(the BigQuery billing export, read keylessly, see below). Browserbase stays grey until BROWSERBASE_API_KEY is in Smith's env. The
Temporal read borrows the account-owner key from .env.local, which is the only key
/cloud/usage accepts today. Replace it with a finance-role service account.
Google Cloud is read with no key (2026-09-29). The propflowai.co org enforces
iam.disableServiceAccountKeyCreation and it stays on, so Smith federates its existing
AWS identity instead. In project propflow-billing (522252817699): Workload Identity pool
smith-aws, AWS provider gera-dev trusting account 194925971150, mapping
google.subject=assertion.arn, and condition
assertion.arn == "arn:aws:iam::194925971150:user/gera-dev". The only principal with
roles/iam.workloadIdentityUser on propflow-cost-reader@propflow-billing.iam.gserviceaccount.com
(which has BigQuery dataViewer + jobUser) is
principal://iam.googleapis.com/projects/522252817699/locations/global/workloadIdentityPools/smith-aws/subject/arn:aws:iam::194925971150:user/gera-dev.
The credential config (no secret) is config/gcp-billing-wif.json in agent-smith;
agent_smith.spend_gcp hands it gera-dev's keys from .env.local. It is pinned to the
IAM user rather than a dedicated role because a role gera-dev assumes has the same trust
boundary. Moving to a role later means changing the condition and that one binding.
Checked 2026-09-29: gera-dev gets a token and lists billing_export; gera-admin, in the
same account, is refused with "rejected by the attribute condition". Live check:
uv run python -m agent_smith.spend_gcp in agent-smith. An empty export shows as a grey
"no data yet" row, never $0.
Reading the signals. "table posted" goes stale when a morning passes without a
post. A vendor that cannot be read does NOT fail the step; it shows as a grey row
with the reason. So a fresh signal means the table landed, not that every row was
readable: read the grey rows in #spending. The step's verdict also appears as the
spending-digest row of the Morning queue table in #agent-smith.
Appendix
- https://github.com/PropFlow-Technologies/agent-harness#readme — the README (no docs declared in SYSTEMS.md yet; add some)
cloud-test (named files, in CI, on demand)
| What it does | Runs ONLY the test files a session names, on that session's pushed branch, in a workflow_dispatch job on the cheap runner (propflowai: vitest; agent-smith: pytest), and prints the verdict and the failing names back in the session. The escape hatch of ADR-0040's amendment (layer 5): with it a session has no reason left to run a test locally, so the guard can refuse every local run. Refuses before dispatching when the branch is not pushed, a named file has uncommitted changes, a file is missing, or (agent-smith) the id is a cross-repo test CI cannot run. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/gera-propflow/local-bin |
| Deployed at | ~/.local/bin/cloud-test → .github/workflows/cloud-test.yml in propflowai and agent-smith (dispatched from main) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → cloud-test (named files, in CI, on demand) |
Liveness signals
| Signal | How it is read |
|---|---|
| called recently | log-match-age · ~/.claude/jobs/cloud-test/calls.log · matches cloud-test repo= · stale after 10080 min |
When a session uses it, and when it must not. The guard admits at most N
test files the branch CHANGED (layer 1) and the default for everything else is
push — CI runs the affected set on the PR and the verdict reaches the session
by push (ADR-0039). cloud-test is the gap between those two: a handful of
specific files, on this branch, now. It is not a suite runner: the workflow
takes a space-separated list of paths, validates the charset, verifies each
exists on the ref, and runs exactly those. One small job per call, 10-minute
cap, #15881's cheap-machine runs-on, no sharding, nothing in ci.yml.
How the verdict gets back. The workflow is dispatched from main (the file
has to exist on the dispatch ref), so its own check suite attaches to main's
commit. The job prints one line, CLOUD_TEST_RESULT state=… passed=… failed=… sha=<checked-out sha>, plus a job summary, and posts no commit status on
purpose: a status lands in the PR's check rollup, which merge gates read
(agent-smith's merge_gate.py), so a flaky red would block a sha whose real CI
is green. The CLI reads that line from gh run view --log and refuses it
unless sha is the commit the caller pushed. The review-webhook listener does
not see this run (verified 2026-10-03: it handles no status and the run is
not on the PR), so the CLI polls gh run view at 60 s — the floor, never
faster, and never gh run watch (3 s polls; a dozen watchers drain the
account's API budget) — for up to 15 minutes, then exits: 0 pass · 1 fail ·
2 refused · 3 could not find out (dispatch/poll error, timeout, a conclusion
that is not a test result, no result line, or a result for another sha) ·
4 superseded, only when a newer run on the same branch proves it (the
workflow's concurrency group is the branch — one running, one queued, 002's
ask in the CI-cost thread); 4 means no verdict, never "fine".
What it proves and what it does not. A green cloud-test is a claim about
the named files on the pushed commit, nothing wider — the PR's own CI remains
the gate. The CI-cost thread (C0C10AS4U15 / 1790993207.945769, 2026-10-03
11:45) saw the proposal before the workflow PRs were armed.
Appendix — everything linked from this entry
tab-reopen (the window a broken pipe hit comes back whole)
| What it does | On the Mac: remembers which Claude session every iTerm tab holds (one file per tab, keyed by the tab's own iTerm id), and after a broken pipe rebuilds the window that broke — its mpf (mini) AND pf (Mac) tabs, broken and still-connected, in the old order — as one new window. Opens straight away when exactly one on-screen window has broken tabs; asks 1/2/3 otherwise. Every tab is `mpf <sel>` / `pf <sel>`, probed first, so it never starts a second copy of a conversation. |
|---|---|
| Kind | reflex |
| Repo | https://github.com/gera-propflow/dotclaude |
| Deployed at | ~/.claude/scripts/session-set.py (writes ~/.claude/tab-map/, launchd co.propflow.session-set every 300 s, Mac only) + _pf_note_tab in ~/.claude/shell/aliases.sh (pf/mpf write the same file at launch) + ~/.claude/skills/tmux/open-top.py (`mpf open`, `mpf open --windows`, `mpf open --window KEY`) + ~/Applications/Reopen Tabs.app (scripts/install-reopen-tabs-app.sh) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → tab-reopen (the window a broken pipe hit comes back whole) |
Liveness signals
| Signal | How it is read |
|---|---|
| tabs recorded (Mac) | file-age · ~/.claude/tab-map · stale after 1440 min |
Why it exists (2026-10-03). mpf open already reopened the mini sessions a
broken pipe dropped, but only by time ("this Mac had a tab to it in the last 60
minutes") and only the mini's. Gera: "can it see both mpf/pf i had open in prev
window and open the same … the pipe breaks and i want a new window with all new
tabs that are connected", and "imagine a bat file in my mac i can click … if
its not sure like which window, i can ask, like 1,2,3 … but if it knows it should
just open."
How it knows the window. Every iTerm tab carries ITERM_SESSION_ID
(w0t3p0:<GUID>) into the ssh / tmux launcher it runs; iTerm reports the
same GUID as unique id of session, along with the window and tab it sits in
now. A tab whose pipe broke keeps its GUID (its shell is still there at a
prompt). So the tab map says what each GUID last held; iTerm's live layout says
where each GUID is; the join is the old window, in order. pf/mpf write the
record the moment a tab starts, so a tab opened a minute before the break is
known; the heartbeat refreshes it every 5 minutes and covers launchers that do
not go through them. The tmux server's own process (tty ??) is never a tab.
When it asks. A window still on screen with at least one broken tab whose
session is still alive in tmux is the case it is sure of. A closed window's
records (seen in the last --within minutes, held by no live client now) are
offered but never opened unasked — a tab closed on purpose looks the same. With
neither, mpf open in a terminal falls back to the older time-based finder;
the app lists that guess and asks. Nothing old is ever closed.
Appendix
- https://github.com/gera-propflow/dotclaude#readme — the README (no docs declared in SYSTEMS.md yet; add some)
Smith Slack socket receiver
| What it does | Receives EVERY Slack event for the Agent Smith app over Socket Mode (the app has no HTTP Request URL, so nothing else in the fleet or in prod receives Slack events) and routes it: Smith's brain, the reply routers, clara_hold_review for fair-housing cards, and hold_reply_relay, which forwards a human reply on a propflowai 'Held for review' card to prod POST /api/slack/events (ADR-0046); and the #alerts watch, which admits Smith's own marked alerts AND third-party Sentry issue posts (`sentry_alert.is_sentry_post`, ahead of the bot-post drop) into `_route_alert` — ack, mute gate, dedup, then the alert decider, which may open a fix PR under its tiered merge gate. A Sentry `AiFallbackError` (a scripted flow's AI takeover) is routed there as a script gap and becomes a script-fix PR in the owning repo (ADR-0056) |
|---|---|
| Kind | actor |
| Repo | — |
| Deployed at | — |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Smith Slack socket receiver |
Liveness signals
| Signal | How it is read |
|---|---|
| receiver loaded | launchd · co.propflow.smith-slack-socket |
| events arriving | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches slack_socket rx · stale after 120 min |
| Sentry alerts admitted | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches slack_socket sentry alert chan= · stale after 4320 min |
| boot resumed open owner-window asks | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches slack_socket boot resume: · stale after 10080 min |
Catalogued 2026-10-07, three months after it shipped (agent-smith #2729, 2026-06-30), and its absence cost exactly what this file exists to prevent. propflowai #9948 built a second Slack inbound (an HTTP Events route) after a grep found "NO existing Slack inbound path anywhere in this codebase", which was true of one repo and false of the system. Because the app is Socket Mode, that route never received an event, and every Slack approve on a held message from 2026-09-19 to 2026-10-07 was dropped. The relay is the fix, and ADR-0046 is the decision. Before you build anything that receives Slack, read this entry. Slack events arrive here and nowhere else.
Appendix — everything linked from this entry
finding inbox (the one-poster hand-off, and whether it strands)
| What it does | ADR-0051 made the thread OWNER the single path to a person: a lane does not post into a human's Slack thread, it DEPOSITS its finding into `thread_findings` (sqlite, WAL, 0600, ~/.claude/smith-state/thread-findings.db, keyed on (channel, thread_root)) and the owner says it on its next reply in that thread. This row is the liveness answer the ADR says that inbox OWES: every 5 minutes a probe READS the store and writes the word `INBOX OK` only while nothing has been waiting longer than an hour — so a stranded hand-off stops being invisible. ⚠️ IT WOULD LATCH WITHOUT AN ACKNOWLEDGEMENT, and the acknowledgement NOW EXISTS — agent-smith#1062 (`299fdcea`) — which is the single most important thing to know before reading the rest of this row, because every note below was written while it did not. The latch it closes: nothing clears a waiting finding except the owner CARRYING it, and some findings can never be carried (over the brevity cap), so one of those would hold this row non-green until it was carried and a SECOND stranding next week would change nothing a reader could see. The release is `smith-finding-inbox --ack <set>`, a compare-and-set over the digest on the line you actually read, recorded in the probe's OWN ledger — so `INBOX OK` means "nothing UNACKNOWLEDGED is stranded" and `acked=N` is the rest of that sentence. `_ack_note` is the design and names the five surfaces to check it by. ⚠️ AND THE READ COMES FIRST, WHICH THIS ROW COULD NOT SAY UNTIL agent-smith#1097 MERGED (`7887b92c`): `smith-finding-read` is the only door in this family that prints a finding's WORDS — the probe reports counts, ages, lanes and a digest and never a word, and `smith-thread-finding` is the deposit door with no read mode — so the order is READ, then CARRY or ACK. ACKING SOMETHING UNREAD IS THE FALSE GREEN THIS DESIGN REFUSES AT EVERY OTHER LAYER: the probe is built so it cannot clear the condition it watches, and `--ack` is a compare-and-set for a finding a person has DEALT WITH, so an ack taken off a `set=` nobody has read buys the green and leaves the findings unsaid. IT IS A HUMAN DOOR — it refuses an LLM session before the inbox is opened — AND ITS LIMITS ARE STATED AT THE COMMAND RATHER THAN COPIED HERE: `finding_read_cli`'s module docstring names residuals (1)–(4), including how far that refusal goes, and they are CITED because a second copy of them in this file would go stale on its own. It MEASURES ONLY. It never deposits, never marks anything posted, never moves a finding between threads and never wakes an owner; `thread_findings` has no delete path and the probe adds none, because a probe that could clear the thing it watches can manufacture its own green. |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/agent-harness |
| Deployed at | `agent_smith.finding_inbox_probe` in the main checkout ~/code/PropFlow/agent-smith, on PATH as `smith-finding-inbox`, scheduled by launchd `co.propflow.finding-inbox` — ARMED ON THIS MINI, and the CHECK rather than this sentence is what settles that, because a date here rots: `launchctl list | grep co.propflow.finding-inbox` prints the label and `scripts/install-finding-inbox-launchagent.sh --status` prints `loaded ✓`. It was rendered from deploy/co.propflow.finding-inbox.plist and bootstrapped by one deliberate operator run of that installer (agent-smith#1049); `_pr` holds WHEN, and holds it ONCE — this field deliberately carries no date, and `_pr` says why. The check above survives what a timestamp cannot: run `--remove` and the date here would go false silently, while `launchctl list` simply stops printing the label. Nothing a merge does loads it, so this is a claim about THIS mini and any other box that has not had that run is still unarmed. The undo is the installer's `--remove` (launchctl bootout, then rm of the rendered plist). See `_pr` for what the first pass printed: ARMED IS NOT GREEN. The store it reads is written by `smith-thread-finding` (agent_smith.thread_findings_cli) from any lane, and read by `agent_smith.activities.slack_send._carry_findings` inside the owning reply workflow. |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → finding inbox (the one-poster hand-off, and whether it strands) |
Liveness signals
| Signal | How it is read |
|---|---|
| probe scheduled | launchd · co.propflow.finding-inbox |
| nothing stranded | log-match-age · ~/.claude/smith-state/finding-inbox/finding-inbox.log · matches INBOX OK · stale after 15 min |
| snapshot fresh | file-age · ~/.claude/smith-state/finding-inbox/finding-inbox.json · stale after 15 min |
Detail document: ~/.claude/smith-state/finding-inbox/finding-inbox.json (the Systems tab opens this row into it).
Where to edit, when these disagree. The JSON notes above are the NEAR copy — they sit beside the signals the evaluator reads — and the prose below is deliberately not a second statement of them: it carries what the JSON does not, namely what a green reading proves, the state-dir trap, the measurement tables, the renderer measurement, and the fact that the gate's signal is still unclaimed. A future editor who finds the two saying the same thing should delete it HERE and keep the JSON.
What a GREEN reading proves here, which is the question ADR-0009 says every
new entry answers in review. Green on nothing stranded proves the probe
opened the inbox and found no finding waiting longer than the window. It does
not prove anything was ever delivered — delivery is mark_posted, and this
row measures the absence of a backlog, not the presence of a hand-off. So the
honest reading is "nothing is stuck right now". On 2026-10-13 that reading was
false for 19.5 hours with nothing on this page to say so.
⚠️ AND GREEN IS NOT REACHED ON THIS BOX TODAY, which a reader should know
before treating a red row here as news. See _pr: the probe's only green word is
INBOX OK, and findings are waiting that nobody can carry without returning to
the Slack threads they belong to. How many is on the line, not here — n= on
the newest INBOX STRANDED line of finding-inbox.log, re-measured every 300 s,
for the same reason _pr gives for not carrying the number either.
⚠️ THE SECOND HALF OF THAT PARAGRAPH CHANGED AND THE CONCLUSION DID NOT. It
used to end "the acknowledgement that would unlatch it is designed and unbuilt".
The unlatch shipped — agent-smith#1062 (299fdcea) — so this row is no
longer red with no way out; it is red with two ways out, and both of them are
a person's to take: carry the findings, or acknowledge them with
smith-finding-inbox --ack <set> quoting the set= off the line you actually
read. ⚠️ AND BOTH OF THEM COME SECOND. Read what is waiting first —
smith-finding-read, agent-smith#1097 (7887b92c) — because neither act is
decidable without the words. job carries the why. A probe may do neither for
itself, by design — one that could clear the condition it reports on can
manufacture its own green. ⚠️ AND THIS ROW NAMES
NO OWNER FOR EITHER ACT, which is a real gap now that the row is active and a
reader is being asked to treat its red as an alarm. Who may carry a stranded
finding is parked for a human; _ack_note cites the decision, and the id is not
re-typed here because this row has had to retract one of those already. Until it
is answered, a reader who finds this red has somewhere to look and nobody to
hand it to, and saying so is better than an alarm that looks owned. Until one of those
happens, this row is red by the design it documents rather than by a fault,
which is exactly the state ADR-0009 says a signal must not quietly normalise.
⚠️ AND THE SILENCE CORRECTED IN job AND IN THE PARAGRAPH BEGINNING
"THE SECOND HALF OF THAT PARAGRAPH CHANGED" IS RECORDED RATHER THAN QUIETLY
FIXED, BECAUSE THE COUNT IS THE ARGUMENT. Measured on this row's own fenced
block at c30acc0, the commit this correction branched from:
24 substring matches of --ack — 15 of them the clearing ACTION and 9 the
--acked ledger flag — against smith-finding-inbox 8 times, launchctl 5,
the deposit door smith-thread-finding once, and zero mentions of any way
to read what a finding SAYS. So the page an operator opens when hand-offs are
stranded was 15-to-0 in favour of making the red go away over learning what they
were dealing with, and the one act it named is the act that is only safe after
the one it did not. The measurement is anchored to a sha rather than to a clock
reading for the reason _ack_note anchors its parser list to one: the number is
this correction's evidence, and a correction keeps its evidence. The row was
not wrong to be silent until now — there was no read command to name, and a
catalogue entry for a command that is not on main is a claim about something
that does not exist. ⚠️ AND A MERGE IS A DELIVERY RECEIPT, NOT AN
OUTCOME: nothing here says anyone has run smith-finding-read, and the
findings this row reports are still waiting, un-acked.
The trap that would have made this row lie, named because it is one line in a
plist. The inbox path resolves through slack_action_store.state_dir(), which
reads SMITH_STATE_DIR at call time and otherwise falls back to a
hardcoded directory. Both armed Smith jobs set it explicitly
(co.propflow.smith-worker, co.propflow.smith-slack-socket — verified with
plutil -p, 2026-10-13), so a probe launched without it would be free to
read a different, empty inbox and write INBOX OK forever: a false green
produced by the instrument rather than by the system. The rendered plist sets
the same value, and every log line and the snapshot name the file that was
actually read.
⚠️ AND THE GUARD THIS PARAGRAPH DESCRIBED DOES NOT EXIST — CORRECTED
2026-10-14 by the adversarial reviewer on agent-smith#1049, which found the same
false sentence copied into the installer. This paragraph went through two
wrong versions. The first ended "so the claim is checkable instead of assumed",
which is a hope about attention rather than a check. Its replacement said
agent-smith#983 "now refuses: an un-overridden pass with SMITH_STATE_DIR unset
writes INBOX UNREADABLE (exit 1)". No code path writes that line for an
unset variable. Measured against main:
agent_ident.state_dir()falls back to~/.claude/<AGENT_SLUG>-state, which for this seat is byte-identical to what the plist sets — so unset is the HEALTHY case, not the dangerous one. (slack_action_store.state_dir()retracts the "hardcoded directory" wording in its own docstring too.)- A presence check on the variable was tried and was removed, and
finding_inbox_proberecords why in as many words: it "was wrong in BOTH directions" — it refused correct unset passes, and it passed every set-and-wrong value, which is the case that actually produces a false green.
So against THAT failure — a state dir that is set and wrong — there is exactly
one guard, and it is --expect-db, which the rendered plist passes: it
compares the RESOLVED inbox against the path the same artifact declares,
and a mismatch refuses with exit 1. The failure it covers is a
variable that is set and wrong — a stale export, a temp dir left from a
control run, a typo in a hand-edited plist — and an inbox that is not there
reports nothing stranded by design, so the green would look ordinary.
⚠️ AND HERE IS WHAT PINS THAT, BECAUSE "TRUST THIS ROW" IS THE PROBLEM THIS
PARAGRAPH IS AN INSTANCE OF. The claim has two halves and they are checked in
different places — neither of them here. ⚠️ And they are TWO tests with similar
names, so the three-layer table BELOW names each by its FUNCTION rather than its
file: a reader looking for the pin on one row could otherwise pick the other.
(An earlier version of this sentence said "above". The table is further down the
page, so the pointer sent a reader the wrong way — a small instance of the thing
this whole paragraph is about.) That the flag's value AGREES with the state dir
the same plist declares is pinned by
test_the_plist_expects_the_inbox_its_own_state_dir_resolves_to, which feeds
the plist's own environment to the real thread_findings.db_path() and
compares the answer to the plist's own
flag, so a rename on either side reddens; its red reading is "the plist expects
…/findings.db but its own SMITH_STATE_DIR … resolves to …/thread-findings.db —
so every pass of the deployed job would refuse with INBOX UNREADABLE." That the
flag then REFUSES a mismatch at run time is pinned by the probe's own suite.
⚠️ THAT CITATION HAS LANDED — agent-smith#1049 merged as b824dbe4, which is
the commit that added tests/test_finding_inbox_deploy.py (checked by name on
that repo's main, not inferred from the PR title). The caveat it replaced is
kept rather than deleted, because the reason for it outlives this instance: the
eighth false claim this row's family produced was a plist comment naming a test
file that pinned nothing — the sentence asserted the pin before the pin
existed. So the rule stands even now that this one is satisfied. A citation
to another repository is this row's own claim until that PR is on main,
verified by reading the code and by nothing that runs in this repository — and
this row says which of those two states each of its citations is in, rather than
leaving a reader to assume the generous one.
Why this correction is worth its space rather than a quiet rewrite: a reader
who hit a false green would have opened the installer, been told there were two
independent guards, and gone looking for an INBOX UNREADABLE line that nothing
writes FOR AN UNSET VARIABLE. The invented refusal was the same defect as the
--ack surfaces above — a document promising probe behaviour that does not
exist — and it reached a second repository because the installer's author trusted
this row.
⚠️ AND THE SCOPE IN THAT FIRST SENTENCE WAS ADDED ON REVIEW OF THIS VERY
CORRECTION, because the unscoped form was the defect one more time:
INBOX UNREADABLE IS a real output — the probe writes it for a store it cannot
read, which _signals_note above has always said — so "a line that is never
written" would have been a FOURTH wrong version of this paragraph, wrong in the
opposite direction to the third. ⚠️ And the first attempt at saying so was put
mid-paragraph, where it stole the antecedent of the sentence after it: "that"
then pointed at the fourth-version remark instead of at the invented refusal.
Named on review. A clause inserted into a finished argument moves what the next
sentence refers to, which is why this one is at the end.
⚠️ AND HERE IS WHAT --expect-db STILL DOES NOT COVER, because this paragraph
has now overstated itself twice. The flag compares the resolved inbox against
the value the deploy artifact declares, so it catches a variable that
disagrees with the plist. It cannot catch the plist being wrong in both places
at once, and it cannot catch a value changed on the box after deploy. Three
layers, each covering a different moment and none subsuming another:
| when it goes wrong | what catches it |
|---|---|
| wrong at deploy (the plist's own state dir) | test_deploy_state_dir.py::test_every_agent_plist_declares_its_own_state_dir — every agent plist DECLARES a state dir and it matches the slug it runs as |
| wrong at run (the env disagrees with the plist) | --expect-db, exit 1 — pinned by test_finding_inbox_deploy.py::test_the_plist_expects_the_inbox_its_own_state_dir_resolves_to, which is a DIFFERENT claim from the row above: not "a state dir is declared" but "the flag AGREES with the one this plist declares" (in flight — see below) |
| changed on the box after deploy, both lines agreeing | nothing, as of agent-smith#1049 — the db= on every log line is the only thing that would show it to a reader. ⚠️ A bare "nothing" is a claim about ANOTHER repository that goes false the day someone adds a check there and tells nobody here, so it is dated rather than asserted |
The third row is the honest residual. Closing it would need a source of truth
for the expected path that is not SMITH_STATE_DIR, and SMITH_STATE_DIR is
the source of truth — which is the same reason the probe refuses to invent a
presence check.
A second footgun closed in the same review, worth knowing before anyone
hand-runs this: --db <a copy> with no --log used to read the copy and
append to the LIVE signal log — and this row's own measurement tables are
exactly such hand-runs. Since the relay matches only the substring INBOX OK
and never reads db=, one of them would have written a green line about an
empty copy and held this row healthy for a whole staleness window. --db now
requires --log and --snapshot together, or --dry-run. Measured 2026-10-14
(an earlier version of this sentence named a third flag, --acked, and said it
did not exist — ⚠️ it does, and it is retracted here rather
than quietly corrected, for the same reason _ack_note keeps the text it
corrects: it names the acknowledgement ledger, and supplying it is itself what
makes a run investigative, which is a STRONGER statement than the one being
corrected rather than a weaker one):
--db <copy> alone refuses with REFUSING to write — --log: this run is
INVESTIGATIVE and it is inside the live state directory, and the same call with
--log and --snapshot pointed outside both protected roots is accepted.
The limit this paragraph used to describe belonged to what was then the
unbuilt half — an unbounded ack ledger holding the row green — and it is left
in _ack_note with the rest of that argument. ⚠️ That half SHIPPED
(agent-smith#1062), so the limit is now a live one and not a design one, and it
cuts both ways on purpose: an acknowledgement is keyed on (finding_key, rowid)
so a finding that recurs after being SAID alarms again, while a stranded finding
re-deposited and still unsaid keeps its rowid and stays acknowledged. The limit
the probe keeps either way is in job: an ack records that a person SAW a
finding, it does not deliver it, mark it posted or remove it, so nothing actually
unlatches a stranding except the finding being CARRIED.
The lines the probe really writes — RE-TAKEN 2026-10-14 FROM THE DEPLOYED
JOB, because this table has now been false in both directions. ⚠️ Its previous
version said "Every INBOX OK row in the previous version carried an acked=
field, and every INBOX STRANDED row a set=. The merged probe writes
neither." It writes both — agent-smith#1062 landed them — so a table
correcting an over-claim became an under-claim one PR later. It had already gone
false once the other way, measured against the #983 branch and correct about
that branch until the branch was split. The lesson is not about either number:
a dated receipt in a document cannot defend itself, which is why every row
below names where its bytes came from, and why the LIVE-INBOX row is now the
deployed job rather than a hand-run.
| input | the line, db=… trimmed for width |
source |
|---|---|---|
| the LIVE inbox, read by the ARMED job | INBOX STRANDED n=4 oldest=43.6h set=b05527bc acked=0 window=3600s lanes=driver-one-thread-voice,driver-vercel-build-spend,emergecy-line |
~/.claude/smith-state/finding-inbox/finding-inbox.log, the signal file itself — ⚠️ A READING AT 2026-10-14 11:51:53 CDT, the job's first pass, and NOT the current line: tail -1 that file for the current ONE, which is a different line and not a re-take of this one. ⚠️ EVERY VALUE QUOTED HERE — n=, oldest=, set=, acked=, the lanes — is a snapshot of that moment and none of them is a standing fact, which is the same instruction _pr gives for the count: read it off the line, not out of this document. |
| a seat with nothing waiting | INBOX OK stranded=0 window=3600s |
relay/tests/fixtures/findingInbox/clear/finding-inbox.log |
| a seat with one finding past the window | INBOX STRANDED n=1 oldest=2.0h window=3600s lanes=fixture-lane-waiting |
relay/tests/fixtures/findingInbox/stranded/finding-inbox.log |
Those are bytes, not transcription. The two FIXTURE rows are committed
fixtures that agentflow-relay#351 generated by running the probe and then feeds
to this catalogue's own evaluateSignal, so the format contract between the
writer and this reader is under test rather than under agreement. ⚠️ They
predate agent-smith#1062 and carry no set= or acked= — they are still
valid for what they pin (the verbs, and this reader's handling of them) and they
are NOT evidence about the fields, which is the exact confusion the paragraph
above is a correction of.
THE LIVE-INBOX ROW is the only one written by the scheduled job against the real store — so, unlike a hand-run, its bytes cannot have been shaped by the flags of whoever took them.
⚠️ The rows are named by CONTENT, NOT BY POSITION. This paragraph once said "the second and third" and "the first row is the strongest of the three", which go wrong the next time a row is added and are the same shape as the retraction tally this row has already had to drop. Those two phrases survive above only inside this retraction, which is why they are quoted rather than reworded.
The discriminator is still the age, and it is still checked — the same store
reads INBOX OK or INBOX STRANDED depending only on how long a finding has
been waiting, so the signal is keyed on stranding and not on findings existing.
That property is pinned in agent-smith's suite against the real module; it is not
re-typed here, because a table in this file is the one artifact nothing runs.
What the previous tables showed, why it is still not here, and what changed:
the acknowledgement walk-through (an ack, a stale set= refused, the next pass
reading INBOX OK acked=2, a later stranding still alarming, an acked finding
recurring) and the mutation that proved the (finding_key, rowid) keying
load-bearing. Both described the then-unmerged slice, and both were deleted
rather than reworded. ⚠️ The slice has since merged (agent-smith#1062), so
that walk-through is now takeable — and it is deliberately NOT taken here. It
would require a real --ack against the live inbox, which would acknowledge
findings a person has not yet seen; acknowledging them to produce a table
row is the mute button _ack_note argues against, reached through this
document. The keying is pinned instead by the recurrence test in agent-smith's
own suite, which runs; a table here is the one artifact nothing runs.
⚠️ A test COUNT is deliberately not quoted here. The sibling PR's own body carries it against the sha it was measured at; a number copied into this row goes stale on that PR's next commit and then reads as a current fact.
⚠️ WHAT THE SIGNALS CANNOT SEE, so a reader does not assume they do: neither
of them reads a NUMBER. nothing stranded is a literal-substring match on
INBOX OK, so n=3 and n=1 are the same non-green state and no signal type in
this catalog can tell them apart; the count is on the line, and the line is in a
file on the mini. The same is true of the acked= the acknowledgement now puts
there, and of set= — no signal reads either — which is one reason _ack_note
keeps the unlatch off the signal and in a deliberate human action.
⚠️⚠️ AND THE DETAIL DOCUMENT'S CONTENTS DO NOT REACH THE SYSTEMS PAGE AT ALL —
CORRECTED 2026-10-13, having first been asserted here without being checked. An
earlier version of this paragraph said a field of the snapshot was something "the
Systems page renders when the row is opened". It does not. Measured in the
renderer:
propflowai/src/app/(workspace)/(operations)/agents/_components/SystemsTab.tsx:351-362
keys a dedicated panel off the system id — ccswitch, agent-specialists,
pacer — and every other entry, this one included, falls through to one line
showing the detail's STATE and its SOURCE PATH (detail: updated 19s ago (…json)) and never its data. The log signal is no better: evalLogMatchAge
renders "INBOX OK" 20s ago — the pattern and the age, not the matched line.
So, stated as it actually is: this row's two signals tell a reader whether
something is stuck and whether the probe is alive, and nothing about WHAT is
stuck. That is in two files on the mini — the lanes named on each
INBOX STRANDED line of finding-inbox.log, and the stranded array in
finding-inbox.json. What would close it, named so the next person does not have
to re-derive it: a finding-inbox branch in that SystemsTab renderer, which is
a propflowai change and not this row's to make.
ADR-0051 says the GATE owes a signal too, and this row does not claim it.
The gate is the refusal in slack_post that routes a lane here; its failure
shape is a different question (a refusal that fires when it should not, or does
not fire when it should), measured on a different surface, and bundling it into
this row would give two conditions one green light. It stays unclaimed, and
saying so here is the point of ADR-0004's liveness clause.
Appendix — everything linked from this entry
propflow-ingest
| What it does | Pulls everything a new customer already has (AppFolio first, then Gmail, Outlook, Drive, Monday) into a per-company intake sheet whose every row carries a reproducible PATH to where the policy was found, and digs deeper until Fede's replay of the customer's past guest-card questions clears the go-live bar, so the customer only reviews the last 10 to 20% |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflow-ingest |
| Deployed at | Mac mini launchd co.propflow.ingest-daily (~/Library/LaunchAgents/co.propflow.ingest-daily.plist), daily 05:30 local: schedule/ingest-daily.sh runs `ingest run --company <id>` for every id in ~/.propflow-ingest/companies, from its own runtime worktree ~/code/PropFlow/.propflow-ingest-runtime at origin/main |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → propflow-ingest |
Liveness signals
| Signal | How it is read |
|---|---|
| daily job scheduled | launchd · co.propflow.ingest-daily |
| a scheduled run passed | log-match-age · ~/Library/Logs/propflow-ingest-daily.log · matches INGEST-DAILY OK · stale after 1800 min |
| an ingest run landed | log-match-age · ~/.propflow-ingest/runs.log · matches ingest org_ · stale after 1800 min |
| a replay run landed | log-match-age · ~/.propflow-ingest/runs.log · matches replay org_ · stale after 11520 min |
Detail document: ~/.propflow-ingest/last-run.json (the Systems tab opens this row into it).
Proposed, and the v1 foundations run (2026-10-15). Gera approved the system on 2026-10-15, from that day's founders' sync with Fede. The decision record is ADR-0084 (2026-11-05). This paragraph used to cite ADR-0059, which is a different decision (the Smith runtime-clone refresh); the 2026-10-15 record was never written here. The repo now holds the policy registry (93 rows), the intake-sheet schema and store, AppFolio connectors (the synced copy in DynamoDB, public listings, the company website), the runner and CLI, and skeletons for the explorer and the replay hook (propflow-ingest #2 to #8).
Scheduled daily since 2026-10-16 (propflow-ingest #29), so active. The
launchd job co.propflow.ingest-daily runs every company in
~/.propflow-ingest/companies (one id per line, outside git: a company id is a
customer identifier) at 05:30 local. Three signals, three different facts, all
on a 30-hour budget (one missed day plus slack):
daily job scheduled: the launchd job is loaded and its last exit was 0.a scheduled run passed: the shim writesINGEST-DAILY OK <company> <verdict>only when the CLI exited 0 AND printed its verdict line; a failure isINGEST-DAILY FAIL <company> rc=Nand never matches. A scheduler that fires and fails therefore reads stale here, not green.an ingest run landed: the CLI's own verdict line inruns.log(ingest <org> <date> coverage=NN% … MET|BELOW_BAR), written on every saved run, scheduled or by hand. Coverage BELOW_BAR is a true reading, not a fault: liveness here means "it ran", the coverage bar is the plan's job.a replay run landed(propflow-ingest #50): every savednpm run replay -- runappendsreplay <org> run=N set=<heldout|dev> against=<sheet|kb> pct=NN% MET|BELOW_BARto the sameruns.log. It is on an 8-day budget because replay is on demand until the weekly scheduled replay lands. The synthetic fixture company (org_fixture-0001) matches too: a fixture run proves the pipeline runs end to end, but it is not a customer score. Runs themselves sit in DynamoDBpropflow-ingest-sheetsasSHEET#<org> / REPLAY#<nnnn>, holding ids and verdicts only, never question text.
The detail block is the heartbeat ~/.propflow-ingest/last-run.json, written on
every pass, failures included: start and finish, the code version, whether the
runtime update worked, and each company's exit code and verdict.
It never logs in. A scheduled pass reads DynamoDB, public pages and AppFolio API reports through propflowai's own client; robot reads go only through the deployed robot's read-only route with its saved session and are skipped when that session needs a login. No stored login is ever decrypted.
What it does not own. It never writes to customer settings: propflowai's onboarding pre-fill reads the sheet and a person clicks. The appfolio-browser-agent stays a transport a connector may use. pms-knowledge-base seeds the "usually lives in" hints. Raw dumps are customer PII and live in per-company S3, never in the repo.
Re-scoped 2026-11-05 (ADR-0084):
the Company knowledge base's app. Gera rebranded ingest as the Company
knowledge base and its grid as the staff view of it. What changes for this row,
part by part of the plan: the rail writer and the authority-first order (part
13c); ROW# becoming evidence-only (13b, on Gera's go); the rename to
kb.propflowai.co (part 14). Its signals stay as they are until then. Three more
are planned, each added in the PR that makes it real, not before (ADR-0004's
liveness clause: a signal with nothing behind it would be a fake green):
- "every live company ran today": the existing daily signal, read against the company list that moves into the knowledge base in 13c instead of the dotfile.
- "a rail write landed in 36 h": a
log-match-ageon the rail writer's run log (13c). - "kb.propflowai.co answered and ingest.propflowai.co redirected": the relay has
no HTTP signal type today, so this one lands with the rename (part 14) as a
file-ageon a probe's result file, or stays prose.
propflow-ingest-find-worker is unchanged. propflow-ingest-audit and the
coverage % read their facts from the rail from the evidence-only cutover (13b).
Appendix — everything linked from this entry
Company knowledge base
| What it does | Holds every company's facts as one free-form entry per topic on the company and building nodes in propflowai (brain.<topic> rows, with where each came from, who confirmed it and whether Clara may say it), read through one fenced reader (readBrain / forClara) by every renter-facing surface and the lease paperwork, written through one writer (writeBrainEntry) by the Settings tab "Knowledge base · Ask", the kb.propflowai.co grid, the CLI twin and the ingest rail |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | not yet: propflowai src/lib/domain/brain/ lands in parts 1a and 2 of the plan; the nightly sweep (part 6) and the daily learning (parts 12 and 18) follow |
| Status | proposed |
| Live on the Systems tab | propflowai.co/agents → Systems → Company knowledge base |
Liveness signals
No liveness signals declared — the catalog renders this system Unknown until one is added. Unknown is not healthy.
Proposed 2026-11-05, and its signals list is empty on purpose. Nothing of it runs yet, and an empty list renders UNKNOWN, never healthy (ADR-0004). The signals ADR-0084 §5 names are planned, each added by the part that produces it:
- "census parity attached to every lane PR" (from part 1b, the census on the bench door).
- "nightly sweep ran" (part 6,
scripts/brain/sweep.ts; the first signal, and the one that flips this row toactive). - "no company over 70% of the knowledge budget" (part 17; the budget already posts to #alerts at 70%).
- "G-BRAIN1 logged N unbacked figures / 0 unexplained" (part 11, the observe-only check).
- "no open question older than N days at a rolled-out company" (part 12, the daily list and its stall page).
- "grid value equals tab answer on the TEST company" (part 13b's contract test).
- "end-state grep green on main" (part 19, the removal lane).
Why its own row and not part of propflow-ingest. The store lives in
propflowai and is read on every Clara turn; ingest is one of its writers and
the staff window onto it. A dead ingest run means the knowledge base serves
yesterday's entries, not that Clara is reading nothing, so the two fail apart.
Appendix — everything linked from this entry
propflow-ingest find worker
| What it does | Claims the queued "Look for it" requests staff post on ingest.propflowai.co and runs one bounded, read-only headless Claude explorer per request, one at a time, which posts its progress and a plain-words found / not-found-because reply into the row's thread; a found value waits for a person's Teach it |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflow-ingest |
| Deployed at | Mac mini launchd co.propflow.ingest-find-worker (~/Library/LaunchAgents/co.propflow.ingest-find-worker.plist), every 60 s and at load: schedule/find-worker.sh runs worker/find-worker.ts --once from its own runtime worktree ~/code/PropFlow/.propflow-ingest-find-runtime at origin/main |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → propflow-ingest find worker |
Liveness signals
| Signal | How it is read |
|---|---|
| find worker scheduled | launchd · co.propflow.ingest-find-worker |
| a worker pass ran | log-match-age · ~/Library/Logs/propflow-ingest-find.log · matches FIND-WORKER OK · stale after 60 min |
Detail document: ~/.propflow-ingest/find-worker.json (the Systems tab opens this row into it).
Why its own row and not a fourth signal on propflow-ingest. The daily run
and the find worker fail apart: one is a 05:30 batch, the other a queue that must
answer within minutes of a click. Folding them would let a healthy daily run
paint a dead queue green, or a quiet queue paint the batch amber.
What "a worker pass ran" proves, and what it does not. FIND-WORKER OK is
written at the end of every pass that reached the web API and walked its queues,
idle passes included (an idle pass takes about a minute). launchd never starts a
second instance while a pass runs, so a pass working a request writes nothing
until it ends: an explorer is capped at 15 minutes and a Run-now ingest at 45,
hence 3600 s rather than a few intervals. The signal goes stale when the job
stops, the web API token stops working, or a pass wedges.
It does NOT prove an explorer reached an answer: a request whose session fails is
closed as not-found with the reason, and shows as a WORKED … not-found line
rather than as a stale signal. That outcome lives in the row's thread on the page.
Guardrails, in the code, not in this prose. The explorer runs claude -p with
--permission-mode dontAsk and an exact allow-list (this repo, public pages, the
web API's read commands, and one progress-line command). It cannot run ingest run, so it never spends the AppFolio request allowance the live sync shares; it
cannot claim, teach or close anything itself, and never logs in or decrypts a login.
Decision record: ADR-0059 (propflow-ingest) and the web-app spec D9.1.
Appendix — everything linked from this entry
propflow-ingest audit
| What it does | On a staff click of Run audit on ingest.propflowai.co, turns a company's latest intake sheet plus its public AppFolio listings page into a customer-ready AppFolio check-up (a grade per section, what they do well first, every number with its period and source), saves it as a new AUDIT#<org>/<n> that is never overwritten, and publishes it ADDITIVELY to share.propflowai.co at a secret id as a draft for Gera |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/propflow-ingest |
| Deployed at | Runs inside the find worker (co.propflow.ingest-find-worker, every 60 s): a queued RUNREQ#<org> with kind=audit is claimed and runs `audit/cli.ts run --company <org> --publish`; publishing goes through audit/share-publish.ts to the Vercel project propflow-share (staging copy: propflow-share-staging) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → propflow-ingest audit |
Liveness signals
| Signal | How it is read |
|---|---|
| an audit was published | log-match-age · ~/Library/Logs/propflow-ingest-find.log · matches done audit= · stale after 43200 min |
On demand, so a 30-day budget. Audits run only when someone clicks Run audit,
so a quiet month is not a fault. The signal matches only FIND-WORKER AUDITED <org> <id> done audit=N: a failed audit (or a publish the guard refused) is
reported as failed and never matches. Its queue health is already the find
worker's a worker pass ran.
The share site guardrail, in the code. share.propflowai.co is a static
Vercel deploy that would REPLACE the whole site if deployed as a folder. The
publisher never does that. It builds a preview from the live deployment's files
by sha plus the one new page, and refuses unless the result is a strict
superset with identical shas. It refuses if the domain moved while it worked,
moves the alias, re-checks every existing page's sha1 on the live domain, and
points the alias back on any mismatch. The live site is saved byte-exact in
propflow-docs share/live/ (audio in R2). Proven first on propflow-share-staging
(propflow-ingest #239).
What it reads and never does. It reads ingest's own store, propflowai's synced copy (read-only) and the public listings page, paced. It never logs in, never decrypts a login, never writes to AppFolio or propflowai, and never sends the report to anyone: Gera reviews the draft and shares it.
Appendix — everything linked from this entry
kb-freshness
| What it does | Weekly: re-scrapes AppFolio's help articles into pms-knowledge-base without ever signing in, and proposes generic propflow-ingest recipe paths as new AppFolio path templates; every change arrives as a needs-review PR a person merges |
|---|---|
| Kind | pipeline |
| Repo | https://github.com/PropFlow-Technologies/pms-knowledge-base |
| Deployed at | scripts/kb-freshness from a detached origin/main worktree at ~/.claude/jobs/kb-freshness/kb (launchd co.propflow.kb-freshness, Sundays 09:17) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → kb-freshness |
Liveness signals
| Signal | How it is read |
|---|---|
| job scheduled | launchd · co.propflow.kb-freshness |
| a weekly run finished | log-match-age · ~/.claude/jobs/kb-freshness/runs.log · matches kb-freshness scrape= · stale after 11520 min |
| help articles re-scraped | log-match-age · ~/.claude/jobs/kb-freshness/runs.log · matches scrape=OK · stale after 23040 min |
Phase 4 item 2 of propflow-ingest. Each run ends with one line in
~/.claude/jobs/kb-freshness/runs.log:
<utc> kb-freshness scrape=OK|NO_SESSION|FAILED|BELOW_BAR articles=N refresh=PR|PR_OPEN|NONE promote=NONE|PR|PR_OPEN.
"Help articles re-scraped" reads DOWN until the first good scrape (then STALE
if the session later expires), and that is the truth. The articles at knowledge.appfolio.com are not public (an
anonymous request gets a 401 "sign in first"), and the scheduled scrape runs
with --no-login: it never signs in to AppFolio by itself. The mini had no
saved help-center session when this shipped, so runs record
scrape=NO_SESSION. A person running scripts/appfolio-scrape-kb --login once
on the mini fixes it until that session expires.
Nothing merges itself. Refresh PRs (kb-refresh/<week>) and promotion PRs
(promote/recipes-<date>) carry the needs-review label, the repo has
auto-merge off, and the job opens no new PR of a kind while one is still open.
Promotion refuses customer websites, Propflow settings, history notes, literal
values and identifier-shaped text; a reviewer's permanent rejections live in
not_generic in appfolio/path-templates.json.
Appendix
- https://github.com/PropFlow-Technologies/pms-knowledge-base#readme — the README (no docs declared in SYSTEMS.md yet; add some)
customer data-health check
| What it does | Every hour, compares what each live customer company and building SHOULD have (the newest closed month's income statement, no gaps in the last 12 months, no months with uncategorised account lines, no unknown account names, the newest closed month's occupancy month-end history) with what is actually stored, and posts to #alerts through the shared alert door (kind customer_data_health, Smith decider marker) only when something is missing, as a streak (once, then one daily reminder); says 'checker broken', never 'all clear', when its own positive controls fail |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | appfolio-sync Lambda, hourly `financials` tick, dispatcher only :: src/lib/domain/data-health/run.ts (runDataHealthCheckOrSayBroken); state DATA_HEALTH#<orgId>/STATE in propflow-prod; message anchors are bake-alert streak rows (data-health, data-health-checker) |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → customer data-health check |
Liveness signals
| Signal | How it is read |
|---|---|
| Smith picked up a data-health alert | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches source=customer_data_health · stale after 2880 min |
Why it exists. In one week three customer-visible gaps went unseen: a company's NOI stuck on an old month because months were hidden by account names the mapping did not know, a company with most of its buildings flagged for a year, and occupancy month-end history that nothing had written since June. Each was present-looking on every green dashboard. The check reads expected-vs-present directly from the stored rows, so it sees the absence itself rather than trusting the writer's own report.
Its positive control. A run is broken, and posts "checker broken… does
NOT mean all clear", when it enumerates no companies or buildings, reads zero
financial months across the whole estate, a reader throws, or its planted-
defect canary (a synthetic building with a known stuck month, gap, flagged
month and no occupancy) is not flagged. A crash of the run itself is posted
too. Both ride their own streak, separate from the problem list.
Appendix — everything linked from this entry
Clara front door
| What it does | Answers every cut-over Clara phone line: forwards the call to ElevenLabs' native endpoint with a <Connect action> back to us, and when ElevenLabs is unreachable, out of credits or the conversation fails, asks the caller what is going on (Twilio speech, no ElevenLabs), rings the company's on-call list for an emergency (then starts the emergency alert chain if nobody answers) and takes a transcribed message for anything else; every such call is a conversation on /conversations and one #alerts card |
|---|---|
| Kind | reflex |
| Repo | https://github.com/PropFlow-Technologies/propflowai |
| Deployed at | Vercel production :: POST /api/twilio/voice/clara (src/app/api/twilio/voice/clara/route.ts); reached only by a Twilio line whose VoiceUrl names it |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Clara front door |
Liveness signals
| Signal | How it is read |
|---|---|
| Smith picked up a Clara front-door fallback alert | log-match-age · ~/Library/Logs/smith-slack-socket.err.log · matches source=clara_front_door_fallback · stale after 10080 min |
Why it exists. On 2026-11-03 the ElevenLabs workspace ran out of credits and every Clara call was hung up on within a second: the lines pointed straight at ElevenLabs, so nothing of ours could fall back. ADR-0083 puts our route in front, proves the post-connect signal live, and sends a failed call to the company's own people, read from the settings it already has.
Appendix — everything linked from this entry
Tenio (fake PMS)
| What it does | A PMS we fully control (API + staff site + chaos switches + a simulated building) that PropFlow's adapters, robot and conformance suite are proven on before a real customer's PMS. Its fleet-side part is tenio-voice: a launchd worker on the mini that writes the words Tenio's simulated people say, with `claude -p` on the Claude subscription |
|---|---|
| Kind | actor |
| Repo | https://github.com/PropFlow-Technologies/tenio |
| Deployed at | Vercel project tenio (scope prop-flow-ai1) → https://tenio.dev; Neon Postgres via the Vercel integration. Worker: ~/.local/share/tenio-voice/index.mjs (installed from tenio main by workers/tenio-voice/install.sh), launchd dev.tenio.voice, config ~/.config/tenio-voice/env (TENIO_URL, TENIO_KEY_FILE = a 0600 file holding a Tenio admin key), log ~/.local/state/tenio-voice/voice.log |
| Status | active |
| Live on the Systems tab | propflowai.co/agents → Systems → Tenio (fake PMS) |
Liveness signals
| Signal | How it is read |
|---|---|
| worker loaded | launchd · dev.tenio.voice |
| heartbeat fresh | file-age · ~/.local/state/tenio-voice/heartbeat.json · stale after 2 min |
Detail document: ~/.local/state/tenio-voice/heartbeat.json (the Systems tab opens this row into it).
Tenio is a fake property-management system PropFlow owns, at https://tenio.dev:
an API and a browser-drivable staff site over one service layer, with fake
companies (the test client Resio Residential) and a fake building (Park
West). PropFlow's PMS adapters, the robot and the conformance suite are
proven against it before any real customer's PMS (plan:
docs.propflowai.co/a/pms-agnostic-architecture-2026-11, stage 1). Fake data
only: +1 000 phones, @example.test emails. Its messenger and broadcasts are
simulated inside Tenio and never send anywhere.
Appendix
- https://github.com/PropFlow-Technologies/tenio#readme — the README (no docs declared in SYSTEMS.md yet; add some)