0055 — A launchd signal reads every domain, and never renders a system not-alive on its own
- Status: Proposed. This ADR decides nothing. It records a measured
defect in an instrument, presents five options, and recommends two of them.
The precedent for a proposed-status record is
PR #318
(ADR-0049). No founder has read this, and no option here is in force.
⚠️ Nothing was touched to produce it: read-only throughout — no label
loaded, unloaded, bootstrapped or kickstarted, no daemon stopped or
signalled, no
signalsblock edited, noSYSTEMS.mdentry added. The evidence below only exists while the instrumented systems are left alone. - Date: 2026-10-14
- Deciders — nobody yet; this is a measurement with options attached. Three
independent measurements, credited where they belong:
- The
op-dock-phase-rowsdriver measured theco.propflow.*/com.propflow.*prefix split, and watched a completetracker-refreshtick in the refresher's own log at 2026-10-14 12:23:40Z → 12:39:56Z (lock taken, re-derive, push rejected, lock released) while that label read absent. That tick is the first proof any dock had that an absent label can sit over a running system. - The
dock-found-rowsdriver measured the per-object and per-distinct-label counts and found the cost in decider blockb91582813. - This lane re-derived every number below from
origin/main, and swept thesystemdomain — which is where the answer was. Both earlier docks probed only the caller's own launchd domain, which is why both read the defect as 26 unexplained absences rather than as one wrong probe.
- The
The rule this breaks, stated by the file it breaks
SYSTEMS.md defines the signal types and then, four lines later, states the
rule that governs them (origin/main at 4f82060, lines 30–35, quoted
verbatim):
signals— how the relay proves it's alive. Types:file-age(path + staleAfterS),log-match-age(last line matchingpatternmust be recent),launchd(label must be loaded and not crash-looping),self(the relay reporting on itself). An empty list renders as UNKNOWN, never as healthy — not proving liveness must never look like proving it.
The launchd type violates the mirror image of that sentence. When the
probe cannot find the label it renders down — not proving death looks like
proving it. ADR-0009's
2026-10-13 amendment already named why that direction is the worse one:
A false green is loud once someone checks; a false red is silent forever, because the page simply stops being read.
This ADR is that amendment's rule one layer further out. 0009's two amendments fixed the evaluator's window and then its stamp parser. This is the probe: not what it reads, but where it looks.
The measurement — one grain, stated
Read from git show origin/main:SYSTEMS.md at tip
4f820605e3182fb4d911f872e1a5ab38cc7e1a94 (2026-10-14T15:30:32Z), probed
against this host between 15:34Z and 15:46Z on 2026-10-14. 74 fenced json
blocks, 0 unparsed.
26 of the 36 distinct
launchdlabels the catalog declares are not in the domain the deployed relay probes. 25 of those 26 are loaded in thesystemdomain. One is in neither.
That is the per-distinct-label grain and every number in this ADR is in it
unless it says otherwise. The other grain, carried here once and never mixed
in: 44 signal objects, 17 probe-visible, 27 not — larger because four
labels are declared by more than one system (co.propflow.smith-worker by
five, co.propflow.smith-slack-socket by three, co.propflow.worker-cleaner
and com.propflow.agentflow-supervisor by two each). The distinct-label grain
counts jobs; the object grain counts mentions.
The instrument was validated against the thing it models. launchctl list <label>, run as this host's user, agreed with the live relay's own verdict on
42 of 42 of the launchd signals it serves (GET /api/systems, fetchedAt
2026-10-14T15:40:47Z), with zero mismatches. Paired control: the four exhibit
labels below return rc=113 Could not find service, while
co.propflow.smith-worker and com.propflow.review-webhook return rc=0 with
a live PID. A sweep that found nothing would otherwise be indistinguishable
from a sweep that did not run.
Two method warnings, both paid for here:
- Read
origin/main, never a working copy. The first dock to measure this read the checkout, which was 20 commits behind, 13 of them touchingSYSTEMS.md(398,946 bytes against 461,394 onorigin/main) — and their count was wrong in a way nothing would have flagged. - A
launchdsignal compares a STALE declaration against LIVE state. The declaration comes from a committed file; the loaded set comes fromlaunchctl. Across that 20-commit gap the probe-visible count was identical at 10 in both docks' readings: the catalog drifted, the machine did not. The two halves of this signal type rot at different rates, which is a structural reason it cannot be made reliable by upkeep alone.
Why the labels are absent — and the decision that already covers it
They are absent because launchctl list only ever sees the caller's own
bootstrap domain, and on 2026-09-27 the fleet's headless jobs were converted
to LaunchDaemons in the system domain so they would survive a logout. Probed
per label with launchctl print system/<label>: 25 of the 26 resolve to a
plist under /Library/LaunchDaemons/, and four were state = running at the
moment of the read (tracker-refresh, puller, watcher-doctor,
operator-heartbeat). For an interval job, state = not running between fires
is the normal idle state the evaluator's own docstring says reads ok.
This is not a new finding and it is not an undecided question. That
conversion is
PR #227,
"ADR-0033: fleet background jobs run in the system domain", whose §5 already
says it in the imperative: "Anything that asks 'is job X loaded' asks both
domains. launchctl list sees only the caller's own domain and never shows a
system job. launchctl print system/<label> does, and it works
unprivileged." The PR carries the evaluator fallback, a deploy.sh fix and
two tests.
It has never merged. Opened 2026-09-28T03:14:48Z — 16.5 days ago, last
touched 2026-10-02T06:29:55Z, 12.4 days idle. All eight checks SUCCESS.
27 bot review comments: 3 GREEN, then 24 YELLOW, each one a re-review
triggered by a merge of main into the branch rather than by a finding. It is
now CONFLICTING / DIRTY, auto-merge unarmed. A second open PR (#226) also
titles itself ADR-0033, so the number collides and docs/adr/ on main has no
0033 at all.
ADR-0004 warns that a decision made in conversation dies in conversation. This one was written down exactly as 0004 asks, and died in an open pull request instead — while the instrument it fixes went on publishing false reds every ten seconds for sixteen days. That is the finding this ADR exists to put somewhere a reader will find it.
The exhibits — a count is a symptom; these are the argument
Each was re-verified for this record. Times are UTC on 2026-10-14.
1. co.propflow.propflowai-main-pull — the signal is inverted on a guarantee
a reader is told to trust. propflowai/CLAUDE.md rule 2 leans on this puller
by name ("every 300s") to claim the shared main checkout is current. The
checkout's own reflog names it: the last eight HEAD moves are each stamped
propflowai-main-pull: Fast-forward, the most recent at 15:25:29Z, nine
minutes before the read (reflog records only moves, so the gaps are quiet
periods, not misses). At 15:34Z that checkout sat 2 commits behind, both
younger than the last pull. And the relay says both things at once on one
row: doctrine-pull renders propflowai main pull loaded → down,
"not loaded", beside propflowai main pull completing → ok, health=ok
3 seconds ago. A three-second-old success marker next to a red that claims
the job does not exist.
2. co.propflow.tracker-refresh — absent, and running, and it derives the
board every Driver reads. Two live processes (refresh-trackers, 58m and 8m
elapsed), its own log written 240s before the read, and launchctl print system/ reporting state = running. The peer dock's log tick
(12:23:40Z → 12:39:56Z) is the independent witness. ⚠️ Honest residual on this
row: two of its three behavioural signals (live deriver running,
dock fresh) were stale at the read, which is the layered design working
as its entry says it should — those layers "fail apart" and name which one
stopped. This ADR claims only that the rotation is running, not that every lane
under it is healthy.
3. co.propflow.fleet-medic — absent, and green on both of its own
behavioural signals at the same instant. The relay's own output:
job loaded → down; tick ran → ok, updated 4m ago;
nothing left unhealed → ok, "medic OK" 3m ago. Measured directly:
heartbeat 275s old, last medic OK line 255s old, against 900s and 1200s
thresholds. Its row renders down. ⚠️ Correcting the brief this lane was
given: it cited five live medic processes; there were none at 15:35Z, which
for a 300s interval job is the normal idle state. The heartbeat and the log
line are the evidence; a process count is not.
4. com.propflow.reviewer-token-rotate — the one row with a real failure,
and the probe is wrong about it in BOTH directions. The briefed claim was
that this job is gone. It is not. The label is loaded in the system
domain, runs = 95, run interval = 7200 seconds, state = running — and
its process (pid 33492, plus one child, reviewer-pool-run-check.sh) has been
alive since 2026-10-05 22:23 local, 8 days 12 hours, on a job that is
supposed to finish every two hours. Its log's last cycle done is
2026-10-10T21:15:18Z: 3.77 days, 30.2× its own 10,800s threshold, and its
three log-match-age siblings all read stale, correctly. So: the launchd
signal renders down for a job that is loaded (wrong), and a
domain-corrected probe would render it ok — running (pid 33492) for a job
that has produced nothing in four days (wrong the other way). Only the
behavioural signals tell the truth on this row. Where that process is
blocked is not established here, and this ADR does not touch it. The pipeline's
own entry says what is at stake: "the job died silently every 2 hours for ~13
hours … and its only voice was itself — dead things don't speak. This signal is
the external witness." The witness worked. It had no reader.
What it has already cost
The red column on the Systems tab is mostly noise. From the live render at
15:40:47Z: 72 rows — 29 ok, 26 down, 15 unknown, 2 stale. In 19 of those
26 down rows, every non-ok signal is a launchd one and every other signal
on the row is ok or unknown: host, fleet-stability, memory-guard,
hooks-pin, pacer, cleaner, cleaner-jobs, wt-merged-notify, slacker,
watcher-doctor, fleet-medic, fleet-bake, fleet-controls,
decision-blocks, decision-ladder, astra-relaunch, hello-email,
operator-heartbeat, doctrine-pull. Worst-wins roll-up
(ok < stale < unknown < down) means one false red takes the whole row, so
pairing a launchd signal with behavioural ones does not help by itself —
fleet-medic is already paired, two green to one red, and still renders
down. A reader who learns that red means nothing is reading correctly, which
is precisely how reviewer-tokens — the one genuinely broken row in that
list — went unread for 3.77 days.
And it has already cost a ruling. Decider block b91582813 (still
state: open) cites "SYSTEMS.md 'reviewer-tokens' (rotate-reviewer-tokens.sh,
launchd, pool claude-pool-reviewer) with liveness signals" as its grounds
for treating the mini as the live system of record. At the moment that verdict
was written (2026-10-09T22:01:18Z) all four of that entry's signals already
evaluated red: the three log-match-age ones at 16,082s against a 10,800s
threshold (1.49× past), and the launchd one absent. The citation reads the
presence of a signal block as a credential for liveness rather than its
verdict — which is the behaviour an unreadable red column produces in a reader.
To its credit the ruling paired the ask ("Fede stops writing AND the mini's
cycling gets confirmed/fixed"). That second half has not happened: 4.74 days
later nothing has banked, so the answer as written now lands on a publisher
whose process has been wedged for 8.5 days.
A separate, smaller defect — and it is NOT the cause
The catalog carries two prefix conventions: 32 distinct co.propflow.*
labels and 4 com.propflow.*. Three of the four com. labels are
probe-visible; com.propflow.reviewer-token-rotate is the absent one, and
nothing matching rotate/reviewer/token is probe-visible under either prefix —
so that signal is not false because of a misspelling. The real cost of the
split is that a reader cannot tell a typo from an absence by looking. The
method warning it produced belongs here: a one-character slip searching
com.propflow.* instead of co.propflow.* returns 4 labels instead of 36
and exits cleanly, which would make a correct count look invented.
Options
- (a) Retire the
launchdtype. Every entry that uses it moves to a behavioural signal. Honest, and it deletes the one signal that answers "is this scheduled at all?" — a system whose script works but is no longer scheduled goes quiet with nothing red. - (b) Pair it. A
launchdsignal must sit beside afile-ageorlog-match-agesignal and may never alone render a system not-alive. Needs a roll-up change, not just a catalog rule: worst-wins is what makesfleet-medicred today with two green signals beside it. - (c) An unreadable label renders
unknown, notdown. The mirror of the empty-list rule, in the evaluator. Moves those 19 rows from a false red to an honestunknown(severity 2, not 3 — it does not fake green, and does not make the row ok). - (d) A reconciler that compares every declared label to
launchctland reports drift. Worth doing for its own sake — it would also catch the reciprocal gap this sweep found: 8 propflow labels are loaded on this host and 7 of them appear nowhere inSYSTEMS.mdby label, against an opening rule that says a system not in the file does not exist. (PR #134 already proposes the enumerate-loaded-jobs half.) It is a report, not a fix: it changes no verdict. - (e) Land PR #227's evaluator half, on its own, off fresh
main. One function (evalLaunchdSystem+parseSystemPrint) and two tests, already written and already reviewed GREEN. Measured here:launchctl print system/<label>costs 4.5 ms per probe against 6.2 ms forlaunchctl list— cheaper than the call it falls back from, which matters because the relay evaluates on its only thread.
Recommended: (e) then (c), as two small PRs, in that order.
(e) first, because it is the only option that makes 25 of 26 reds correct
rather than merely quieter, the code exists, and it is the decision already
recorded in ADR-0033 §5 — no new authority is needed, only a merge. It should
go as its own PR carrying the evaluator and its tests and nothing else: #227
bundles SYSTEMS.md, deploy.sh and a new catalog entry across five files,
and that breadth is why it collected 24 merge-of-main re-reviews and ended
conflicting. The deploy.sh half needs sudo -n and belongs to whoever owns
deploys; it must not block the one-function fix.
(c) second, because it is what keeps this from recurring. (e) fixes the one
domain we know about today; (c) makes any future domain, sandbox or host the
probe cannot read fail to unknown instead of down. That is the catalog's own
stated rule, applied in the direction it was never applied in.
(b) is a good rule and is not sufficient alone — exhibit 3 is the proof: it
is already paired, and still red. Adopt it only together with a roll-up change
that says a lone launchd red cannot carry a row. (a) is rejected as the
primary fix: exhibit 4 shows the opposite failure (a job loaded, running, and
producing nothing) and the answer there is to trust the behavioural signals that
already caught it, not to delete the scheduling check. (d) is advisory and
independent of all of the above.
The residual this ADR does not close
- 26 labels are probe-absent. This ADR establishes the state of four of
them. For the other 22 it establishes only that 21 are loaded in the
systemdomain and 1 (co.propflow.finding-inbox) is in neither — not that any of their systems are doing their jobs. Loaded is not working. No clean bill of health is implied for any row here, and no claim that any of them is down. co.propflow.finding-inboxis the one declared label neither domain knows. Its entry is newer than the catalog the deployed relay serves, so its rendering could not even be read today. Whether that system is dead, unarmed or hosted some third way is unverified here.- Why exhibit 4's process is wedged is unverified, deliberately: diagnosing it means touching it, and it is the live evidence for this record. It needs an owner's decision, which this ADR does not make.
- This record is one host's reading. Everything was measured on the mini.
Why a new ADR and not a third ADR-0009 amendment
ADR-0009 is the right home for the rule, and its two amendments were both
written in place, in the same PR as the fix they recorded — that is the
precedent (8c2407f) and it is a good one. It does not fit here: this document
fixes nothing and decides nothing (the lane that wrote it is forbidden to
touch the instrument, which is what keeps the evidence intact), and an
amendment in place would assert as settled a rule nobody has chosen. When an
option here is chosen and built, the implementing PR should fold the rule into
ADR-0009 as its third amendment and leave this file as the measurement it cites.