Two days · one fleet · ten wrong diagnoses Every claim here was measured, not reasoned · 2026-08-27 → 08-28
Ten wrong answers, and the one shape they shared
Every failure was an instrument that answered a nearby question — and each looked more careful than the thing it replaced.
LESSON · THE CHECK HAS TO COME FROM OUTSIDE THE THING BEING CHECKED
docs.propflowai.coFLEET LEARNINGS
02 /VERDICT
Before the mechanism — what it actually cost
Nothing here was a hard problem. Every hour went to believing a measurement that was answering something else.
Not one of these was solved by thinking harder about the failing instrument. Every single one was caught by a different instrument.
LONGEST WRONG TRAIL
4h
three diagnoses, all off a viewer process
GUARDS SILENTLY OFF
13
one escaped quote disarmed all of them
ALREADY DOCUMENTED
Jul 15
the 4-hour lesson, written 6 weeks earlier
NOW ENFORCED
6
hook rules that block it instead of describing it
The recurring cost is not ignorance. It is a correct-looking reading of the wrong thing.
measured on the mini · 88 panes then, 92 the next day02
03 /THE PATTERN
Said once, because everything else is an instance of it
An instrument answered a nearby question, and the answer was read as the one we asked.
WHAT WE ASKED
The real question
"Which account is this session billing?" · "Is this account capped?" · "Did the model hear the warning?" · "Does this file read the variable?"
WHAT WE MEASURED
The nearby question
"What does this pane hold?" · "Did the request fail?" · "Did the hook run?" · "Does this file contain the string?"
The tell: when a fix reads as more rigorous than the thing it replaces, check that it did not also narrow what it can see. Three of today's did exactly that.
the shape, not the incidents03
04 /THE PATTERN
Same shape, five different subsystems, one day
Each one looked like the careful choice.
01 · PROCESS
ps eww on a tmux pane
Answered "what does this pane hold", not "what is the session using". The pane was a viewer. Cost: 4 hours.
02 · API
A 429 status code
Answered "did the request fail", not "is the account capped". A refused request and a real cap are the same status.
03 · HOOK
systemMessage
Answered "did the hook run", not "did the model hear it". Every non-blocking warning was invisible for a day.
04 · REGEX
\bTOKEN\b
Answered "is TOKEN a word", not "is TOKEN present". _ is a word character, so it never matched inside the variable name.
05 · SCAN
"Is this inside quotes?"
Answered "is it quoted", not "is it inert". A double-quoted string still expands — and \" does not close it.
each caught by a DIFFERENT instrument — never by re-reading the failing one04
05 /CASE 01
The most expensive one, and the most ordinary
We measured a window onto the session, not the session.
A tmux pane running claude attach <id> is a viewer. It makes no API calls. Reading its environment describes a process that is not the one being rate-limited.
What we did three times
Read the pane's credential, diagnosed, respawned the pane. Each respawn replaced the viewer; the real session never restarted.
Where the session actually was
A daemon spare, a different pid entirely, with no bearer at all — authenticating as a completely different account.
What broke the loop
Not more analysis. A fresh pair of eyes with no investment in the existing theory.
Two questions from Gera killed three theories: "how are all other sessions working then?" and "but you're on Opus... no". Both were about evidence the theory could not explain.
Two of these four states are the same status code.
200 · 13 HDRS
Healthy
A real reply, and the usage meters ride along.
429 · 13 HDRS
Genuinely capped
This request would exceed your account's rate limit.
429 · 0 HDRS
Refused — not capped
Error. This is what you get without the Claude Code system prompt. Indistinguishable from a cap by status alone.
401 · 0 HDRS
Token revoked
The only state that actually means "re-mint".
Our health check read every 429 as "capped". Because the refusal was unconditional, a perfectly healthy fleet would have reported as fully exhausted — on every account at once, which reads like an outage rather than a broken gauge.
ccswitch#14 · caps are PER-MODEL: a Haiku 200 says nothing about Opus06
07 /CASE 03
The one that could have caused real damage
One escaped quote disarmed every safety rule after it.
Our command guard blanks quoted text so that writing about a dangerous command cannot trigger it. It treated \" as closing the quote. Everything after that read as a live command — or didn't read at all.
FALSE ALARM — visible
Prose got blocked
A message merely describing a dangerous command was refused. Annoying, obvious, harmless.
SILENT — the dangerous half
Real commands went through
A genuine destructive command, after one escaped quote, was allowed. The guard had stopped guarding and looked exactly like a clean command.
And it was in both test harnesses. The scripts we were using to verify the rules had the same defect — so their reliability varied with the punctuation of the case being tested.
pre-existing · found while auditing an unrelated change07
08 /CASE 04
The sequel, 2026-08-28 — same shape, two fresh instruments
Exit 36 does not mean the keychain is locked. It means the session is wrong.
macOS scopes login-keychain access to a security session. Every SSH login lands in Background — and so does the tmux server started over SSH, every Claude process inside it, and therefore every ccswitch hook. security refuses all of them with errSecInteractionNotAllowed = exit 36, the same code a genuinely locked keychain returns.
THE GAUGE
kc_read_status → "locked"
Answered "can I read this from here", not "is this keychain locked". So write_active concluded the keychain governed and could not be written — and for 20 hours every rotation logged outcome: keychain-locked and changed nothing, while ccswitch best correctly named an account at 2% of its weekly.
THE CONTROL
The same call, from a different session
security find-generic-password … → 36. sudo launchctl asuser 501 … → 0. Keychain untouched in between. One command settled what a day of reasoning had not.
THE SECOND ONE
rotate-bearer: "43% of its 7-day cap — nothing to do"
Answered "is the weekly budget spent", not "can this token serve a turn". The active bearer was 429 with its 5-hour window at 100%. Three separate caps; the picker read one.
Then the fix reached nothing. Every tmux session inherits CLAUDE_CODE_OAUTH_TOKEN from the tmux server environment, and that overrides the keychain entirely — so the account switch succeeded, verified clean against the bank, and the fleet never felt it. Two rotation systems on one box, neither aware the other exists.
ccswitch#31 · 5h / 7d / per-model are SEPARATE caps08
09 /WHY
The uncomfortable finding
The four-hour lesson was already written down — six weeks earlier.
The viewer-versus-session distinction was documented on 15 July, reinforced after an incident on 24 August, and the general lesson sat in the same file under a heading that named it exactly.
Why it didn't reach us
It was filed by subsystem — under session cleanup. The work that day was authentication. The right note was read, and only the line being looked for was taken from it.
The real failure mode
Knowledge that requires you to already suspect the answer in order to find it. A perfect note nobody retrieves is worth nothing.
What actually changes behaviour
Something that interrupts the wrong action at the moment it is taken, without anyone having to remember to look.
This is not an argument against documentation. It is an argument against documentation as the only control for a mistake that is cheap to detect automatically.
"if a fact has to be remembered to stay true, it will be false"09
10 /THE FIX
Six rules that block or warn, instead of a page that explains
Each one is a mistake we actually made, priced.
BLOCK
Respawning a viewer pane
Today's four-hour bug. Now refused, with the real session's location in the refusal.
BLOCK
Killing a viewer process
Measured 24 Aug: killing 66 viewers ended roughly 39 hosted sessions. The prediction at the time was the opposite.
BLOCK
Two sessions on one transcript
Resuming a session id that a live process already holds.
WARN
Reading a credential from the shell
Tells you what a new process would get, not what this session is using.
WARN
Reading a detached pane
Renders zero bytes. Empty means "nothing was drawn", not "the session is dead".
A warning that the model never sees is not a warning. Ours emitted a field that does not reach the model — every non-blocking rule was firing into the void until it was measured.
~/.claude/hooks/guard-bash.sh · 69-case matrix, live pids10
11 /THE FIX
Four habits, each bought with a specific hour
Rules that would have caught today, on the day.
01
Run the control through a known-positive input
A verifier reported "all combinations passed" while rendering zero slides. Only pushing a known-good file through it — 18 vs 0 — exposed it.
02
Break the fix and watch the test go red
A test that passes with the fix and without it proves nothing. Two of today's did exactly that before being rewritten.
03
Say what you did not check
A skipped case and a passing case looked identical in the summary line. Silence about coverage is how a shrinking suite keeps reporting green.
04
Derive the list; never hand-maintain it
CI named five test files by hand. There were eight. The three added later had never run — and a missing test looks exactly like a passing one.
every rule here is a specific incident from 2026-08-2711
12 /SHIPPED
Merged, deployed to both machines, verified after
The gauges now discriminate, and the traps now block.
THE GAUGES
ccswitch #14 · #15
The health probe can tell a refusal from a cap, and probes the model the fleet actually runs. Account rotation no longer moves onto an exhausted account, or picks by filename. And the pool alert stopped leading with the one action that logs other people out.
THE TRAPS
6 hook rules · 69-case matrix
Three destructive actions blocked outright, two warned, and the quote-scanner bug that disarmed all of them fixed. Every case runs against live processes, with a control proving each assertion fails without its fix.
One finding worth its own line: the token bank showed nine accounts and holds six. Two pairs are the same account banked twice — including one minted that day specifically to add headroom. It added a file and no capacity.
both machines current · CI now discovers its own suites12
13 /CLOSE
If nothing else from this deck survives
The check has to come from outside the thing being checked.
Not one of these ten wrong answers was caught by looking harder at the instrument that was lying. Every one was caught by a different instrument, a fresh pair of eyes, or a control that already knew the answer.
When a fix reads as more careful
Check that it did not also narrow what it can see. Three of today's did — and each looked like the rigorous choice.
When a guard goes quiet
Quiet is the same shape as working. Prove it still fires before trusting that nothing is wrong.
When you already documented it
Ask whether it can be enforced instead. A note only reaches someone who already suspects the answer.
The good news: none of this required being smarter. It required measuring the thing we were actually asking about.