Built from tonight's incidents, live Verified against the running mini · 2026-08-29
Which account is this session actually spending?
Sessions, bearer tokens, and how a pool of Claude accounts gets composed into one working harness — reconstructed from a real outage and a real fix, tonight.
VERDICT · two credential stores that don't talk, and a process that can't change its mind
docs.propflowai.co01
02 /VERDICT
Lead with the answer
Every incident tonight traces to the same two facts.
A session's credential lives in one of two stores that do not share state, and a running process's environment is fixed the moment it starts. Everything else — the outage, the herd, the wedge — is a consequence of forgetting one of these two things.
CREDENTIAL STORES
2
keychain pool · bearer bank — independent
PRODUCTION PRs BLOCKED
5+
one overloaded account, same 30-min window
SESSIONS ON ONE ACCOUNT
10
before tonight's anti-herd fix landed
RATE-LIMIT AXES THAT MATTER
2
5-hour AND 7-day — the worse one wins
A process cannot be told to switch accounts mid-flight. Every fix in this deck is either about choosing better before exec, or detecting that exec already went wrong.
ccswitch · rotate-reviewer-tokens.sh · this session, 2026-08-2902
03 /SESSIONS
First, the vocabulary
A "session" is four things wearing one name.
01 · ID
Session id
A UUID that outlives everything else — the one stable handle across restarts, renames, and crashes.
02 · PROCESS
The claude process
Has a pid. Dies on --resume, on a crash, on a manual restart. A new pid every time, same session id.
03 · HOST
The tmux pane
Where the process actually runs. Inherits the tmux server's environment at the moment the pane was created — not the shell's.
04 · RECORD
The transcript
~/.claude/projects/<cwd>/<sid>.jsonl — the only thing that survives every restart intact. Every fact in this deck was read from one.
Everything that goes wrong tonight goes wrong because one of these four moves and the others don't find out.
~/.claude/projects/*/<sid>.jsonl03
04 /CREDENTIALS
The root split
The keychain pool and the bearer bank don't know about each other.
Keychain pool
What ccswitch switch touches
Lives in the macOS keychain. Revocable — a login can be killed from the Anthropic side. This is what /login manages, and what a normal Claude Code session authenticates with by default.
Bearer bank
~/.claude/.setup-tokens/*.token
Minted by claude setup-token. One-year lifetime, scope user:inference, revocation-immune — nothing on the Anthropic side can kill it early. Handed to a process via CLAUDE_CODE_OAUTH_TOKEN.
A session running on a bearer is invisible to every keychain-based health check. Tonight's stuck session showed "ACCT LOGIN DEAD — rotating" while running on a perfectly healthy bearer — the statusline was reporting on a store the session wasn't even using.
ccswitch: bearer_path() · local_bearer()04
05 /CREDENTIALS
A third file, and the trap in its name
Most sessions run on a file that forgot whose it is.
The real bearer
~/.claude/.setup-tokens/propflow.evals@gmail.com.token — the email is in the filename.
What most sessions get
~/.claude/.tokens/account7.token — a numbered copy, rewritten every 2 hours by a launchd job.
What's lost
The filename no longer says which account it is. Nothing about the copy can identify itself.
The fix
Match sha256(token)[:12] against the bank's own index.json — the token proves its own identity; the filename never has to.
This is why the anti-herd fix (slide 13) can't just read a filename to know which account a session is spending — it has to hash the actual token.
ccswitch: bearer_email() — sha256[:12] match05
06 /MECHANICS
The one fact under everything
A process's environment is decided at exec. Nothing after that can change it.
tmux pane opens
→
claude execs, reads env
→
credential now frozen
→
rotation happens elsewhere
What rotation CAN do
New
Point the next process at a healthier account — the symlink, the keychain, the tmux server's default env.
What it CANNOT do
Live
Change what an already-running process is authenticating with. The only way in is killing and re-execing it.
tmux server env wins, too.CLAUDE_CODE_OAUTH_TOKEN set at the tmux server level overrides the keychain for every pane on that server — a ccswitch switch can report success and reach nothing.
ccswitch: do_switch() · bearer_publish()06
07 /DISCRIMINATION
Two 429s that look identical and aren't
The exact system prompt string is load-bearing.
Missing the identity lines
Headerless 429
Drop the system: "You are Claude Code…" string or the anthropic-beta: oauth-2025-04-20 header, and a valid sk-ant-oat token is soft-blocked. Body is literally "Error". Zero rate-limit headers.
A genuine cap
Real 429
Same status code. Carries 13 rate-limit headers — the account really is capped, and the headers say by how much.
Get this wrong and every slot in a probed pool reads dead — the whole roster looks exhausted when nothing is wrong. This is the single easiest way to misread the harness.
/api/oauth/usage — the endpoint built for exactly this question.
What a bearer gets
403 permission_error. Scope user:inference cannot call it. No exception.
The only door left
The rate-limit headers riding on every real /v1/messages response — anthropic-ratelimit-unified-5h/7d-utilization and -status.
Net effect
Every health check in this deck is really just one throwaway request, read for its headers — the content of the response is never used.
ccswitch: _rl() · bearer_usage()08
09 /MEASURED
Measured tonight, not assumed
The same token, probed with four different models, reports the same numbers.
CLAUDE-OPUS-4-8
0.96
util_7d
CLAUDE-OPUS-5
0.96
util_7d
CLAUDE-SONNET-5
0.96
util_7d
CLAUDE-HAIKU-4-5
0.96
util_7d
The unified 5h/7d headers are subscription-wide, not per-model. Re-confirmed independently by a second session probing the same accounts an hour later.
What the probe model actually decides: whether the request gets rejected (a model-scoped cap 429s it) — never what headroom comes back. Two different questions; one easy to conflate.
token_probe() run 4×, same token, 2026-08-29 ~03:2xZ · corroborated by a second session at 03:38Z09
10 /INCIDENT
Tonight's outage, in one line
An account answered 200 at 96% of its weekly window — and it was made primary.
What the old check asked
"Does it answer?"
A 1-token probe returning HTTP 200. propflow.lab passed this cleanly — and became the reviewer's primary account.
What actually mattered
"Does it have room?"
util_7d: 0.96, status_7d: allowed_warning — riding on that same 200, unread. It hit its wall hours later.
propflowai#6514 — both primary AND fallback produced no verdict. 5+ other open PRs hit the identical signature in the same window. The reviewer's own picker asked the same wrong question of the same endpoint — having a picker didn't help, because both layers read aliveness and called it capacity.
A 429, or a status marked rejected, always sinks — but never disappears.
01
98% free on both axes, one axis rejected
Sinks below everything. A number is not consent — gera@propflowai.co answered 200 with status_7d: rejected and ranked last of 8 tonight.
02
Capped (429), but the credential is valid
Kept — demoted, never dropped. A pool that refuses everything leaves CI with no reviewer at all, which is worse than a reviewer on a nearly-spent account.
03
Unmeasured — no headers came back
Sorts behind every measured account. Not proving headroom must never be worth more than proving a little.
rotate-reviewer-tokens.sh: rank_pool() tiers12
13 /THE HERD
Built, tested, correct — and doing nothing
The anti-herd picker computed the plain ranking, because nothing ever registered a session.
SESSIONS REGISTERED
0
for as long as the feature existed
WEIGHT TERM WITH ZERO REGS
score/1
identical to no weighting at all
SESSIONS FOUND ON ONE ACCOUNT
10
propflow.evals, live, tonight
TESTS THAT PASSED THROUGHOUT
All
by feeding counts directly, never through the real path
pick_for_session weights each account by score/(sessions+1). With zero registrations that term is score/1 for every account — it computes pick_bestexactly. Testing a picker proves the rule; it cannot prove the rule is ever reached.
ccswitch: pick_for_session() · reg_counts()13
14 /THE HERD
The failure that never raises an alarm
A weighted candidate compared to a raw incumbent can never win.
The candidate's score
score / (sessions_on + 1) — always smaller than raw, by construction.
The incumbent's score
Left raw, unweighted, in the first draft.
What happens
On any account with sessions already on it, the hysteresis check can never pass.
How it reads
"No rotation was warranted." Forever. Nothing logs a failure — it looks like a decision, not a bug.
The fix: divide the incumbent's score by max(1, sessions_on) too — the session asking the question is already counted on it.
ccswitch: cmd_auto() — act_score scaling14
15 /IDENTITY
Back to slide 3 — the pid was never the right handle
Keying a registration on a pid double-counts every restart.
What a pid survives
The life of one process. That's it.
What a session survives
Every --resume, every crash, every manual restart — same session id, new pid.
The bug this causes
A rotation's own restart re-registers the same session under a new key. The account it just left away from still shows the old count for up to 30 minutes.
The fix
The Stop hook already carries the session id in its payload. Use it — pid is the fallback only for a human at a bare shell.
All 20 test suites were quietly writing to production.
What looked sandboxed
CCSWITCH_DIR / HOME
Every other piece of state — cache, config, token dirs — is a file, and redirecting the file paths looked like it redirected everything.
What actually needed its own override
The lease TABLE, the pool SECRET
Not files. Don't move with HOME. A test that called reg_put for real wrote a live row — and every suite passed throughout.
The guard built to catch this had its own bug first: it checked a 120-character window that spilled onto the next line, so a table pointed straight back at production still passed. Fixed by reading each value on its own line.
ccswitch tests/ · #37 · tests/no-prod-writes.py16
17 /COMPOSITION
"Composers" — read here as: how the pool gets assembled
Three layers compose into one usable account, none of them optional.
the bank tokens + index.json
→
the ranker min(5h,7d), eligibility
→
the registry who's already on it
→
one assignment per session, per turn
Drop any one layer and the others silently compensate — badly. No ranker: the first alphabetical 200 wins (slide 10). No registry: the ranker is honest but blind to the herd (slide 13). No bank hygiene: the ranker reasons about a token that was never the one running (slide 5).
This term wasn't in the original brief with a fixed definition — treated here as the assembly of bank + ranking + registration into one decision, not a specific named Claude Code feature.
ccswitch + rotate-reviewer-tokens.sh, composed17
18 /IN PROGRESS
What tonight's incident is turning into
A wedged session has a structural signature, not a prose one.
The symptom
Same turn, same result, no progress — a session retrying "continue" into an identical 429, unattended, for hours.
The signature
An assistant message whose entire content is one text block starting with "API Error:" — structural, not a string match on prose that could change wording.
Why the account being healthy doesn't help
10 sessions on one account can trip a short-term throughput throttle that the 5h/7d weekly headroom never sees.
The fix in progress
Detect N repeats within a window, from the session's own transcript tail; restart in place via the same respawn primitive rotation already uses; cap the attempts so a bad respawn can't loop forever.
this session, 2026-08-29 — build in progress at time of writing18
19 /WHAT WORKS
Not everything in this deck is a defect
The parts of the harness that held up under real load tonight.
01 · RESTART
In-place pane respawn
A rotation reaches a running session by killing and re-execing its exact pane with --resume — idle panes now, busy ones at their next natural stop, never forced mid-turn.
02 · DISCIPLINE
Negative controls on every fix
Every rule in this deck shipped with a test that proves the mistake would have been caught — reverting the fix and confirming the suite goes red, not just that it's green today.
03 · DEMOTE, DON'T DROP
A capped account never vanishes
The one design choice that shows up three separate times in this deck (slides 11, 12, 16) and never had to be re-argued.
ccswitch: auto_restart_panes() · negative-control pattern across #34–#3719
20 /CLOSE
The harness doesn't fail loudly. It fails by answering the wrong question.
"Does it answer?" is not "does it have room?" "It's running" is not "it's making progress." Every fix here just replaced one question with the other.