0009 — A liveness signal must fail when the SYSTEM fails, not when its log stops

• Status: Accepted
• Date: 2026-09-01
• Deciders: the ccswitch session, writing up the design decision made while adding the host entry to the catalog (agentflow-relay#92, local-bin#58). The reviewing bot named the omission and it is right: this reasoning would otherwise have lived only in a merged PR body, which is exactly the death ADR-0004 describes. No new authority is claimed here — this records how signals are chosen, and supersedes nothing.

The problem, paid for in a panic

On 2026-09-01 at 13:06 the Mac mini kernel-panicked after 81 hours of uptime. 88 Claude Code processes held 114.8 GB resident on a 64 GB machine; free memory was 14.9 MB, the file cache had been evicted to 1.8 MB, the compressor held 35.6 GB across 65 swapfiles, and the pageout path asked for 3,090 pages and reclaimed 23. Userspace could not be scheduled, watchdogd missed its check-in for 90 seconds, and the kernel panicked deliberately.

Every rung of ADR-0003's ladder was working throughout. ccswitch healed wedged sessions, rotation spread credentials, nudge woke parked sessions — each rung restarting something inside a machine that no rung could see.

The obvious fix is a host entry in SYSTEMS.md. The trap is how it gets its signal.

The rule

A signal must go red for the failure the system actually has. Concretely, when choosing a signal type, ask: what does a green reading here prove?

file-age proves only that something wrote to a file recently. For a sampler, that is proof the sampler ran — never proof of what it found. Had the host entry used file-age on its snapshot, the Systems tab would have rendered the mini Healthy at 14.9 MB free, because the sampler was faithfully recording the machine's death every five minutes. Green would have meant "the thermometer works."

So a signal over a measuring system buckets on a word that only appears when the measurement is good:

• the sampler writes HOST OK only when every threshold passes, and HOST PRESSURE with named reasons otherwise;
• the signal is log-match-age on HOST OK;
• one signal now covers both failure modes — a dead sampler (no lines at all) and a degrading host (lines still flowing, none of them OK).

This is the same shape reviewer-tokens uses for cycle done and nudge for NUDGED, and it generalises: match on a word that means success, never on the absence of a word that means failure. Absence is indistinguishable from the log not being written, from the pattern being renamed, and from the check never running.

The corollary: pick the metric that moves first

Thresholds have the same failure mode as signals — a metric can be true, prominent, and still useless as a warning.

Free-memory percentage is the trap. At panic, inactive stood at 12.2 GB: nominally reclaimable, enough to read as ~19% available on a machine seconds from death. The kernel asked for 3,090 pages of it and got 23. Inactive memory is a promise, not a reserve.

The metrics that actually moved first were swap and the compressor, so those carry the thresholds and free memory is the last resort rather than the headline. When choosing what to threshold, prefer the quantity that degrades early and monotonically over the one that is easiest to read.

Consequences

• Every new SYSTEMS.md entry answers one question in review: what does green prove here? If the honest answer is "that the log is being written", the signal is wrong and the system needs a success word to match on.
• A system that only measures should be kind: pipeline and should say in its prose that it takes no action. Whether work is worth continuing stays rung 3's judgement under ADR-0003; a sampler that starts killing things has quietly promoted itself a rung.
• This does not weaken the empty-signals rule: a system declaring no signals still renders unknown, and unknown still never renders healthy. This ADR covers the next failure along — a signal that is present, green, and meaningless.

Amendment, 2026-10-13 — the mirror rule: a signal must not fail when the system is FINE

• Status: Accepted, amending this ADR in place (the precedent is 8c2407f, which amended ADR-0030/0040/0047 rather than minting new ones for the same decision).
• Decided by: the spend-alarm liveness lane, while correcting the signal that agent-smith#957 was about to falsify. ADR-0004 requires this to be written at decision time, in the same PR as the change.

What this adds

The rule above asks what a green reading proves. Two failures live on the other side of it, and both were paid for on this one row.

1. A PROXY read after it stops tracking its PROPERTY is not a weakened signal, it is a FALSE one. spend-alarm carried a post recorded today, a file-age on the dedupe ledger a Slack post is recorded in. It stood in for is the check alive? and it worked only because of a property of the OLD behaviour: every UTC day the check posted at least once, so the ledger was touched daily. agent-smith#957 severs that deliberately — Fede ruled 2026-10-12 for major things only, so a quiet day is MEANT to post nothing and silence becomes the healthy state. The signal would then have reddened every quiet day and sent someone chasing an outage that is the new intended behaviour. The remedy is to retire the proxy, not to widen its staleAfterS: widening keeps a signal whose green means nothing and whose red is noise, and a quiet WEEK reddens it anyway. So: when a change makes silence healthy, every signal that counted output is now false, and the PR that makes the change (or the PR that catches it) retires them.

2. A signal must go red for the failure the system HAS, and green when it has none — the second half had no rule, and the gap cost weeks of red. The same row's check completing is the right shape by this ADR's own prescription: a log-match-age on cost_anomaly done, a word written only when the run COMPLETES (EXIT_INCOMPLETE raises above it). It still could not carry liveness, because the evaluator's window, not the signal, was wrong: evalLogMatchAge tailed a fixed 64 KB of a log the whole Smith worker writes to. Measured 2026-10-13 — 0 matches at 64 KB, 128 KB, 256 KB, 512 KB and 1 MB; one match at 2 MB, 76 minutes old against a 4 h budget. The marker left the window ~5 minutes after each 3-hourly fire, so the row read down for ~175 of every 180 minutes while the check was completing on schedule, and nobody chased it because it failed toward red.

That is the cost of the asymmetry. A false green is loud once someone checks; a false red is silent forever, because the page simply stops being read. On the day this was measured the fleet showed 27 down signals, and this row was one of them.

The rule: a log-match-age window must span the pattern's cadence at the log's write rate. It is now derived rather than declared — the tail doubles from 64 KB until the pattern is found or the window reaches back past that signal's own staleAfterS, so a declared budget is honoured instead of being silently clipped to log volume. A number in a signal that the evaluator cannot actually honour is the same defect class as a green that proves nothing.

And the rule holds only UP TO THE CAP, which is why the cap must not be the thing that decides. LOG_TAIL_MAX_BYTES exists for a log with no parseable stamps (no horizon to find) and for a pathological file. Set too low it reintroduces exactly the false red this amendment removes: at the 8 MB first draft, spending-digest — 26 h budget, last marker measured 12.49 MB from EOF — would still have read down, which review caught. It is 32 MB, chosen so that the largest budget any catalog entry declares (93600 s) at the busiest log's measured rate (~0.8 MB/h, ~21 MB) is inside it, i.e. so the HORIZON decides for every signal in the catalog. A budget that needs a bigger cap is not a cap problem — it is a signal pointed at a firehose, and the fix is its own log (ci-runs-on-canary is the precedent). A down that stopped at the cap now says so in its detail, because a false red must be diagnosable and the fixed tail's no "X" in recent log was not.

The cost is on the relay's only thread, so it is paid in index scans, not allocations. The evaluator reads synchronously and the relay is single-threaded; splitting a 20 MB window into lines to find one of them would allocate ~300k strings on every 10 s catalog pass. So the match, the matched line's bounds and the window's oldest stamp are all found with bounded indexOf / lastIndexOf scans over the same buffer. A signal whose pattern sits in the first 64 KB — which is every signal on a quiet log — pays exactly what it paid before.

Inventory of the old surface (ADR-0004 §2)

Three catalog entries had written workarounds for one instrument bug. That is the tell this belonged in the evaluator, not in a fourth entry's prose.

Consequences

• Retiring a liveness signal is a catalog change like any other: a reviewed PR against SYSTEMS.md, with the reason recorded ON the entry (a _retired_signal_note, so the next reader does not re-add it).
• spend-alarm now declares ONE liveness signal plus worker running, and says in prose what nothing there proves: no signal on it proves an alarm FIRED. Post-#957 there is no daily artifact to prove delivery with — the check's receipt is a Temporal run row in DynamoDB, and no signal type the relay has can read it. That ceiling is stated rather than papered over.
• spending-digest reads the same log with the same shape and was down for the same reason; the evaluator fix repairs it without a catalog edit.

Amendment, 2026-10-14 — the same mirror rule, one layer lower: the PARSER

• Status: Accepted, amending this ADR in place (the precedent is 8c2407f, and the amendment above it did the same for the window).
• Decided by: the spending-leak-fence lane on the spend-ramp-alert dock, while finishing what agentflow-relay#343 started. ADR-0004 requires this to be written at decision time, in the same PR as the change — see "Is this an ADR-0004 decision?" below, which judges it rather than assuming it.

What this adds

The amendment above fixed the WINDOW: a log-match-age signal could not find its marker because the tail was too short. One layer below the window is the PARSER, and it had the same shape of defect — a signal must be able to read the logs the fleet actually writes, and ten rows proved it could not.

LOG_STAMP accepted exactly one stamp shape: YYYY-MM-DD HH:MM:SS, space-separated, host-local. Measured on the live relay and on disk, 2026-10-14:

test-inbox is the row that makes the stacking visible. Its marker sat 533 KB behind EOF, so before #343 it read down — no "test-inbox exit=" in recent log. #343's widened window finds it, and the row then lands on this parser and reads unknown instead. Two bugs in series: fixing the window alone moved the row from one permanently-wrong state to another, and no reading of the page before the deploy would have shown that.

Why this is the mirror rule and not a nitpick

unknown is the HONEST state for "I could not read it". The evaluator was behaving correctly and saying so; the defect is that ten rows were PERMANENTLY stuck in it — eleven once #343 deploys — with no reading of any system able to change them. A page a third of whose rows can never be green is a page nobody reads — and that bill came due the same night: the nightly turnover benches returned === VERDICT: RED === in both logs, "synced":0,"failed":2 and "synced":0,"failed":4, zero of six work orders syncing back, and nobody was told, because that true red sat beside three broken thermometers.

So the rule, stated at the parser's layer: the set of stamp shapes the evaluator accepts is a CONTRACT with every log writer in the fleet, and a writer that does not know it exists will break it silently — the row goes unknown, every test of the writer stays green, and nothing anywhere fails. Three entries in SYSTEMS.md had already written a workaround for this one parser (red-main, fleet-deploy, tracker-refresh each chose file-age and cited the parser as the reason), which is the same tell the window fix left: a workaround written in a third entry's prose means the bug is in the instrument.

The widening, and the one risk in it

Accepted now: YYYY-MM-DD, then T or a space, HH:MM:SS, optional fractional seconds, optional zone (Z, ±HH:MM, ±HHMM).

The zone is HONOURED, not discarded, and that was the whole risk. A regex capturing only HH:MM:SS and dropping a trailing Z hands new Date() a zone-less string, which ES2015+ reads as HOST LOCAL — so on this UTC-5 mini every …T…Z line would be dated five hours in the FUTURE, and a marker six hours stale would read ok. That is strictly worse than the unknown it replaces: a false green over a dead specialist. Measured against that exact mutant: a 6-hour-old …Z marker on a 4-hour budget reads ok, detail "test-inbox exit=" 60m ago. A zone-LESS stamp (the space form, and a bare T) stays host-local, unchanged — 52 signals read ok from space-form logs today, which is EVERY ok log-match-age signal in the catalog (counted off the live /api/systems response, not estimated), and reading those as UTC would redden the page wholesale.

NOT accepted, as a scope call and not an impossibility: a stamp wrapped in brackets ([2026-10-14T05:49:53.753Z] event.in, which ~/.claude/services/review-webhook/listener.log writes). The ^ anchor is what makes the line opens with its own stamp checkable, and relaxing it to skip a leading delimiter is a different decision about what a log line is. review-webhook's three signals stay unknown after this change; that is named on its entry and pinned by a test, so it stays a decision rather than becoming an oversight.

THE SECOND DEFECT THE FIX EXPOSED, and the reason it is in the same PR

Widening the parser turned two rows from a permanent unknown into a permanent false GREEN — this ADR's ORIGINAL failure, not its mirror. specialists.log is one file carrying 24 tools whose names nest, and the evaluator matches SUBSTRINGS:

While the parser was space-only this was invisible: all three read unknown regardless. updater is leading-space-anchored now ( updater exit=), the form every other signal on this log already used. A latent false green that only a widening can expose belongs in the widening's PR, because the widening is what arms it.

⚠️ AND THE FIRST DIAGNOSIS OF TWO OF THOSE THREE ROWS WAS WRONG — RETRACTED HERE, NOT SOFTENED

For one round of agentflow-relay#349 this section said optimizer and finisher would read down "because nobody has run either tool in a month", and offered a human the choice between status: superseded and a wider budget. Both the reason and both options were wrong, and the correct answer was neither.

~/.local/bin/optimizer and ~/.local/bin/finisher are symlinks — to work-optimizer and dock-finisher, inodes 706783833 and 706783830, created Sep 13 00:40. And _record_run writes os.path.basename(os.path.realpath(__file__)) — realpath, which RESOLVES the link — so an invocation through the alias executes the target AND LOGS AS THE TARGET. Measured by loading each script through its ALIAS path with CLAUDE_JOBS_DIR redirected, so nothing touched the live log: invoked as optimizer it wrote work-optimizer exit=0; as finisher, dock-finisher exit=0; as cleaner, worker-cleaner exit=0. The control that makes that a measurement rather than a tautology: updater, which is NOT a symlink, wrote updater.

The bare optimizer and finisher lines in the log are real — 51 and 213 of them — and they stop at 2026-09-14T05:59:02Z and 2026-09-13T22:31:10Z. They stop because local-bin changed _record_run from abspath to realpath at 2026-09-14 ~01:30 CT, after fleet-grader spent a full cycle reporting work-optimizer: no run in 3h while the aid had just run it twice through that alias. The incident is written up in ~/.local/bin/tests/specialist-log-names-the-canonical-tool.py, which derives the alias list from the checkout and asserts the canonical name is recorded. That fix collapsed the two identities deliberately, so the string those two rows matched can never be written again.

So the widening did not uncover a month of disuse. It converted a signal that was unmatchable-as-unknown into one that is unmatchable-as-down — the same defect wearing the other colour, which is why "red forever is the same disease as unknown forever" was the right instinct attached to the wrong cause.

The fix is the string. Both patterns name what the binary writes now ( work-optimizer exit=, dock-finisher exit=), and both rows read ok — work-optimizer ran 2026-10-14T05:47:19Z, dock-finisher 06:37:49Z. cleaner is the named precedent (§2) and needed no change: it is also an alias entry (~/.local/bin/cleaner -> worker-cleaner) and has always matched worker-cleaner exit=.

Rejected, and why neither was right: status: superseded would assert a retirement that has not happened — both entry points work and are documented; a wider staleAfterS would hide a red this ADR's own amendment refuses to hide. A third option, collapsing each pair into one entry named for the binary, was considered and NOT taken: id is this catalog's machine key and its header says never to rename one casually, deployedAt pointing at the alias is correct because the alias is the documented entry point, and cleaner already establishes the alias-entry-with-canonical-pattern shape. Inventing a second idiom for a solved problem is what §2 forbids. That collapse would be an ADR-0004 decision; this correction is not — it changes no relationship between systems, it makes a pattern name the string the tool already writes.

One marker now carries two rows, and that is stated on both entries rather than left to be noticed. dock-of matches the same dock-finisher exit= string, deliberately and as a PROXY: dock-of writes no specialists.log line of its own (verified — no _record_run, no write site), and every board reader resolves its dock through it, so a fresh dock-finisher run is evidence a board reader resolved a dock. For finisher the same line is DIRECT evidence. Two rows that will always agree, for two different reasons.

The lesson that generalises, which is why this is in the ADR and not only in a PR body: a log-match-age pattern is a claim about the string a binary writes, never about the name anyone invokes it by. An alias makes those two different, and nothing in the catalog can tell them apart — the row looked correct, the tool was healthy, and the signal was dead. When a row is keyed on a tool's name, the check is basename(realpath(<the entry point>)), run against the box.

Is this an ADR-0004 decision? — judged, not assumed

Yes. ADR-0004 asks whether a decision changes how systems RELATE. Widening a parser sounds internal, and if the stamp format were private to the relay it would be. It is not: it is the contract between the catalog reader and every log WRITER on the box — local-bin's specialists, agent-smith's fleet_deploy.py, dotclaude's operator-heartbeat. Those writers were constrained to one stamp shape BY this parser; agent-smith carries a test, test_log_line_stamp_is_what_the_relay_can_parse, that asserts the constraint against the regex itself. That constraint is relaxed now, and a writer in another repo cannot learn it from a merged PR body. (That test still passes: the space form is still accepted. Nothing in another repo has to change — but what is PERMITTED there did.)

The converse was considered and rejected: branch (b) of the fork, making the specialists write the space form. It is more invasive (every writer in three repos), it leaves 77,400 existing lines unreadable, it fixes nothing for hooks-pin, driver-setup or red-main, and it would have to be re-litigated by every future writer. Branch (a) is one regex, one repo, one deploy.

Consequences

• The authoritative statement of what the evaluator can date is LOG_STAMP in relay/src/systems/catalog.ts. The five places in SYSTEMS.md and the one in ADR-0024 that restated the regex, or gave "the parser cannot read ISO" as a reason, are pointers now — restating a regex is scheduling the next stale claim, which is the same lesson #343's "three entries wrote a workaround" table records one layer up.
• Four of those five entries chose file-age partly for a reason this change falsifies. None of them is re-pointed at a log here: each kept file-age on a reason that outlives the parser (status.json is written on EVERY sweep; the rotation's terminal line appears on only ONE path). Re-pointing a signal is a catalog decision with its own evidence, not a side effect of a parser fix.
• hooks-pin and driver-setup are repaired with no catalog edit, the same way spending-digest was by the window fix.
• Nothing on the page changes until ./deploy.sh runs. #343 is merged and undeployed as this is written, so the deploy that lands it lands this too, and the before/after above is the patched evaluator run against the live logs — not a reading taken off the deployed page.

Amendment, 2026-10-26 — when the SUCCESS WORD cannot be written, the signal reads the measurement itself

• Status: Accepted, amending this ADR in place. The precedent is 8c2407f and the two amendments above it, which did the same for the window and the parser rather than minting a number for one decision's next layer.
• Decided by: the four-green-witness dock's reviewer-pool lane, on row the-reviewer-pool-has-five-liveness-signals-and-none-fires-when-every-account-is-capped. ADR-0004 requires this at decision time, in the same PR as the change.

What this adds

This ADR's original rule prescribes a remedy as well as a diagnosis: a signal over a measuring system buckets on a word that only appears when the measurement is good. reviewer-tokens is the first entry where that remedy was not available, and the gap let the catalog's worst false green stand for three hours.

Measured. On 2026-10-26 the code-review bot was unusable across three repos from roughly 00:52Z for about three hours: every account in claude-pool-reviewer answered HTTP 429. Six finished propflowai PRs plus local-bin#666 and agent-harness#1477/#1484 sat with no verdict, each surfacing per-PR as Review infra failure (tool_crash) and All 3 Claude account slots are OUT OF BUDGET (HTTP 429). The row was green throughout. Its five signals are three log-match-age markers, one launchd probe and a file-age on ~/.claude/jobs/reviewer-pool/status.json, and every one of them measures whether the ROTATOR RAN. Off the rotator's own log, five consecutive cycles from 2026-10-25 20:19 to 2026-10-26 04:33 local: each logged reviewer cap … 429 three times, wrote a fresh status document, and closed with cycle done. A rotator that runs perfectly and faithfully records an outage satisfies every freshness signal there is — which is this ADR's original sentence, one layer up from the host sampler it was written for.

Why the prescribed remedy was unavailable, judged rather than assumed

The obvious fix is this ADR's own: make the rotator write a success word and match on it. That is also agentflow-relay#431's freshest precedent — it gave pr-autoqueue a stuck census ok mode=live line and declared a log-match-age on it, needing no new type and no ADR. It was considered first and rejected on three measured grounds:

1. The nearest existing marker is pointed at the wrong pool. The rotator's terminal line is cycle done: N token(s) banked, F above reserve, and in rotate-reviewer-tokens.sh N is the reviewer pool (real) while F is the FLEET pool (fresh, over ranked ← RECORDS ← ACCOUNTS). Through the outage it read 3 token(s) banked, 14 above reserve, and the sibling ALERT: no account above the … reserve on both axes — whose own comment says it exists because "healthy" counted 200s and "stayed silent through the very outage it exists to announce" — never fired, for the same reason. A guard written against this exact disease missed this exact outage because it measures a different pool. That is a dotclaude defect, reported with this amendment and not fixed by it; what matters here is that no marker on that log can carry this predicate today.
2. A substring cannot carry an arithmetic predicate. log-match-age asks "does this string appear, and how old is it". "At least one of five accounts is live" is a count over a document. Encoding it as a word means the writer computing it — i.e. a second repo, a cutover, and a vocabulary that then lives in two places.
3. The measurement is already written down. The per-account state has been recorded and rendered on this very row since 2026-10-22. ADR-0009's remedy exists because a sampler's finding was unavailable to the catalog; here it is sitting in a JSON document the row already declares as its detail.

So the rule this amendment adds: when a system already PUBLISHES its measurement in a document the catalog reads, the signal reads the measurement — it does not ask the system to re-encode it as a word. Matching on a success word stays the remedy of choice wherever the finding is not otherwise published, because it needs no schema contract; pool-live is the narrower case, not a replacement.

The one new type, and its two fail-closed halves

pool-live (relay/src/systems/catalog.ts, evalPoolLive) is green iff at least one entry in the document's accounts array has state ok.

The predicate is an ALLOWLIST. The row asked for "neither capped nor failed", which is the same predicate today and the wrong shape tomorrow: a blocklist reads a newly-invented state as healthy, which is this row's own defect wearing a new word. POOL_LIVE_STATES is {ok} and an unrecognised state is not live.

And the mirror rule decides every non-answer, in both directions:

account status written is deliberately kept beside it rather than replaced: a fresh document with no live account is a capped pool, a stale document is a stopped rotator, and one signal covering both could not tell a reader which it was looking at.

Is this an ADR-0004 decision? — judged, not assumed

Yes, and for the same reason the parser amendment was. The state vocabulary — ok | capped | no-token | failed — is minted by the jq in dotclaude's rotate-reviewer-tokens.sh and is now READ as a predicate by agentflow's evaluator. It was a presentation detail of a rendered table; it is load-bearing now, and a writer in another repo cannot learn that from a merged PR body. The authority stays the script: POOL_LIVE_STATES names the one word it turns on, and the catalog entry's _signals_note names the script as the authority while carrying the enumeration only as a dated snapshot — the shape #431's review rounds settled on for exactly this, after its own prose was caught asserting another repo's constants as standing fact.

And this contract already has a SECOND reader, which is a consequence rather than a caveat. propflowai carries its own copy of the signal-type union (src/lib/domain/agentflow/types.ts) and, in src/lib/domain/agentflow/specialty-jobs.ts, splits the types into WORK evidence (file-age, log-match-age) and PRESENCE (launchd, self), then derives the SpecialtyJobsCard's health from the work bucket alone. Measured 2026-10-26: reviewer-tokens declares a launchd signal, so isBackgroundJob selects it, and pool-live — the only signal on the row that reads a measurement at all — would land in presenceSignals and be excluded from that card's verdict. The /agents Systems tab is unaffected (it renders signalResults worst-wins, over every signal). So this amendment's rule has a corollary it cannot enforce from this repo: a new signal type is not finished when the relay serves it — it is finished when every surface that CLASSIFIES types has learned it. That is a propflowai change, named on the entry and handed to the dock as its own row rather than deferred to a PR nobody has planned.

This is also the first signal type whose predicate is the CONTENT of another system's document rather than its freshness, its mtime or a log line. That is the relationship change, and it is bounded on purpose: the type reads one array, counts one field, and makes no call (ADR-0003 — rungs integrate by signals, never calls). It stays read-only, like every evaluator here.

Consequences

• Every new entry already answers "what does green prove here?" That question now has a second half: if the honest answer is "that the measurement was taken", the signal is still wrong, even when the thing taking it is alive and writing. A pipeline whose subject can fail independently of the pipeline needs a signal per subject.
• A row whose system publishes its own verdict document should declare pool-live beside its file-age, not instead of it.
• Nothing on the page changes until ./deploy.sh runs, and relay-deploy will NOT do it for this change. The relay serves from a gitignored dist/, so merged and deployed are separate states here; the 15-minute relay-deploy sweeper fast-forwards the live branch only when every incoming file is one the running process reads at request time, and HOLDS the moment main carries code that needs a build. A new evaluator in catalog.ts is exactly that, so this one waits for a hand ./deploy.sh.
• The dotclaude defect in §1 above, and the propflowai type-classification ceiling above, are both reported to the dock as their own rows and are not fixed in this PR: one concern each, and each in a different repo's program.