0009 — A liveness signal must fail when the SYSTEM fails, not when its log stops
- Status: Accepted
- Date: 2026-09-01
- Deciders: the ccswitch session, writing up the design decision made while
adding the
hostentry to the catalog (agentflow-relay#92, local-bin#58). The reviewing bot named the omission and it is right: this reasoning would otherwise have lived only in a merged PR body, which is exactly the death ADR-0004 describes. No new authority is claimed here — this records how signals are chosen, and supersedes nothing.
The problem, paid for in a panic
On 2026-09-01 at 13:06 the Mac mini kernel-panicked after 81 hours of
uptime. 88 Claude Code processes held 114.8 GB resident on a 64 GB machine;
free memory was 14.9 MB, the file cache had been evicted to 1.8 MB, the
compressor held 35.6 GB across 65 swapfiles, and the pageout path asked for
3,090 pages and reclaimed 23. Userspace could not be scheduled, watchdogd
missed its check-in for 90 seconds, and the kernel panicked deliberately.
Every rung of ADR-0003's ladder was working throughout. ccswitch healed wedged sessions, rotation spread credentials, nudge woke parked sessions — each rung restarting something inside a machine that no rung could see.
The obvious fix is a host entry in SYSTEMS.md. The trap is how it gets
its signal.
The rule
A signal must go red for the failure the system actually has. Concretely, when choosing a signal type, ask: what does a green reading here prove?
file-age proves only that something wrote to a file recently. For a
sampler, that is proof the sampler ran — never proof of what it found. Had the
host entry used file-age on its snapshot, the Systems tab would have
rendered the mini Healthy at 14.9 MB free, because the sampler was
faithfully recording the machine's death every five minutes. Green would have
meant "the thermometer works."
So a signal over a measuring system buckets on a word that only appears when the measurement is good:
- the sampler writes
HOST OKonly when every threshold passes, andHOST PRESSUREwith named reasons otherwise; - the signal is
log-match-ageonHOST OK; - one signal now covers both failure modes — a dead sampler (no lines at all)
and a degrading host (lines still flowing, none of them
OK).
This is the same shape reviewer-tokens uses for cycle done and nudge for
NUDGED, and it generalises: match on a word that means success, never on
the absence of a word that means failure. Absence is indistinguishable from
the log not being written, from the pattern being renamed, and from the check
never running.
The corollary: pick the metric that moves first
Thresholds have the same failure mode as signals — a metric can be true, prominent, and still useless as a warning.
Free-memory percentage is the trap. At panic, inactive stood at 12.2 GB:
nominally reclaimable, enough to read as ~19% available on a machine seconds
from death. The kernel asked for 3,090 pages of it and got 23. Inactive
memory is a promise, not a reserve.
The metrics that actually moved first were swap and the compressor, so those carry the thresholds and free memory is the last resort rather than the headline. When choosing what to threshold, prefer the quantity that degrades early and monotonically over the one that is easiest to read.
Consequences
- Every new
SYSTEMS.mdentry answers one question in review: what does green prove here? If the honest answer is "that the log is being written", the signal is wrong and the system needs a success word to match on. - A system that only measures should be
kind: pipelineand should say in its prose that it takes no action. Whether work is worth continuing stays rung 3's judgement under ADR-0003; a sampler that starts killing things has quietly promoted itself a rung. - This does not weaken the empty-signals rule: a system declaring no signals still renders unknown, and unknown still never renders healthy. This ADR covers the next failure along — a signal that is present, green, and meaningless.