0003 — The supervision ladder: recovery by rungs, integration by signals
- Status: Accepted
- Date: 2026-08-29
- Deciders: Gera (2026-08-29, on whether ccswitch should be folded into the Temporal/Driver work: "how I'm thinking about it is just like a machine that works cohesively, but also having parts is fair… not everything has to be forced into one monolithic experience"). This ADR is the written answer.
The rule
Recovery is a ladder. Each rung restarts the cheapest thing in its own scope, and escalates only on repeated failure — never reaching sideways or down:
| Rung | Restarts | Owner | Reach |
|---|---|---|---|
| 0 | the turn | Claude Code itself | every session |
| 1 | the process (same session, same work) | ccswitch wedge-heal | every session; self-only; 3/day budget |
| 2 | the credential (and, since ccswitch#39, its per-session shard) | ccswitch rotation | every session |
| 3 | the session (respawn against a goal, judge progress) | Driver / Supervisor | only sessions with a stated goal |
| 4 | nothing — decides | Gera, via decision pages | everything |
Rungs integrate by signals (files, events), never by calls. ccswitch never depends on Temporal or the relay; the Driver never reaches into credential handling. When rung 1 exhausts its budget it emits (a log line, a state file); rung 3 consumes that if a Driver owns the session, and the ops alarm ledger catches it otherwise.
Why the layers stay separate
- Blast radius. If reflexes called the control plane, a Temporal or relay
outage would take down credential rotation and wedge recovery everywhere —
including the majority of sessions that have no Driver at all. Measured
the night this was decided: the wedged session
592da2d2showed○ no operator; only the hook layer could reach it. - Cost class. Rungs 1–2 run inside hooks on every turn boundary of every
session; they are bounded in single-digit seconds (
REG_NET_TIMEOUT,HOOK_NET_TIMEOUT) because a lost heal is a later retry while a lost turn is the user's. A durable-workflow round-trip does not belong on that path. - The kubelet precedent. A node restarts its own crashed containers locally; the control plane schedules and judges. Nobody asks the API server for permission to restart a container, and nothing here asks Temporal for permission to rotate a token.
What this rules out
- Wrapping ccswitch operations in Temporal workflows "for persistence" — persistence at rungs 1–2 is files on disk plus the DynamoDB pool that already exists.
- A Driver (or any agent) mutating credentials, tokens, or the shard map — that state has one writer per artifact (the single-writer rule the pool learned on 2026-08-08, three outages in one night).
- Reflexes making "goal" judgments. Whether work is worth continuing is rung 3's question; rung 1 only asks whether the session is able to continue.
Consequences
- "Should X be integrated into Y?" now has a test: same rung, maybe; different rungs, signals only.
- Every rung's liveness is independently visible on the Systems tab
(
/agents?tab=Systems), fed bySYSTEMS.md— which is also where a new system must declare which rung it lives on (kind:reflex / orchestration / actor / pipeline).