0003 — The supervision ladder: recovery by rungs, integration by signals
- Status: Accepted ⚠️ VOCABULARY: the role this ADR calls the Operator is the Driver since 2026-09-10 —
operatoris the product's word for a PM company. Decision text below is unchanged; ids, binaries, trigger phrases and theOperatorTaskWorkflowtype that still sayoperatorare live invocations and deliberately lag. See ADR-0021. (Appended toStatuson purpose: this adds NO line, so every line-number citation into this file stays true.) - Date: 2026-08-29
- Deciders: Gera (2026-08-29, on whether ccswitch should be folded into the Temporal/Operator work: "how I'm thinking about it is just like a machine that works cohesively, but also having parts is fair… not everything has to be forced into one monolithic experience"). This ADR is the written answer.
The rule
Recovery is a ladder. Each rung restarts the cheapest thing in its own scope, and escalates only on repeated failure — never reaching sideways or down:
| Rung | Restarts | Owner | Reach |
|---|---|---|---|
| 0 | the turn | Claude Code itself | every session |
| 1 | the process (same session, same work) | ccswitch wedge-heal | every session; self-only; 3/day budget |
| 2 | the credential (and, since ccswitch#39, its per-session shard) | ccswitch rotation | every session |
| 3 | the session (respawn against a goal, judge progress) | Operator / Supervisor | only sessions with a stated goal |
| 4 | nothing — decides | Gera, via decision pages | everything |
Rungs integrate by signals (files, events), never by calls. ccswitch never depends on Temporal or the relay; the Operator never reaches into credential handling. When rung 1 exhausts its budget it emits (a log line, a state file); rung 3 consumes that if an Operator owns the session, and the ops alarm ledger catches it otherwise.
Why the layers stay separate
- Blast radius. If reflexes called the control plane, a Temporal or relay
outage would take down credential rotation and wedge recovery everywhere —
including the majority of sessions that have no Operator at all. Measured
the night this was decided: the wedged session
592da2d2showed○ no operator; only the hook layer could reach it. - Cost class. Rungs 1–2 run inside hooks on every turn boundary of every
session; they are bounded in single-digit seconds (
REG_NET_TIMEOUT,HOOK_NET_TIMEOUT) because a lost heal is a later retry while a lost turn is the user's. A durable-workflow round-trip does not belong on that path. - The kubelet precedent. A node restarts its own crashed containers locally; the control plane schedules and judges. Nobody asks the API server for permission to restart a container, and nothing here asks Temporal for permission to rotate a token.
What this rules out
- Wrapping ccswitch operations in Temporal workflows "for persistence" — persistence at rungs 1–2 is files on disk plus the DynamoDB pool that already exists.
- An Operator (or any agent) mutating credentials, tokens, or the shard map — that state has one writer per artifact (the single-writer rule the pool learned on 2026-08-08, three outages in one night).
- Reflexes making "goal" judgments. Whether work is worth continuing is rung 3's question; rung 1 only asks whether the session is able to continue.
Consequences
- "Should X be integrated into Y?" now has a test: same rung, maybe; different rungs, signals only.
- Every rung's liveness is independently visible on the Systems tab
(
/agents?tab=Systems), fed bySYSTEMS.md— which is also where a new system must declare which rung it lives on (kind:reflex / orchestration / actor / pipeline).