What went wrong with the agent fleet Friday night into Saturday, what it cost, and what we fixed by lunchtime.
Window: Fri ~8pm → Sat ~5pm, Chicago time · a record, not a to-do list
The fleet's "is this agent still alive?" check trusted one attendance list, and that list showed one agent when fourteen were working. So it kept signing death certificates for drivers that were very much alive.
Acting on that bad check, the supervisor shut down and replaced the board's driver five times in under two hours, each time killing that driver's whole crew of helpers with it. On the fifth strike it gave up on the job entirely and tried to raise an alarm — but there was nobody on the other end. The driver it "shelved" was in fact fine and drove all night with no supervision, next to a replacement that was working the same board without knowing the first one existed. Come morning that driver stopped itself — on a "hold above 70% of the account" rule nobody gave it — and the safety gate that had been refusing to let it quit gives up after six refusals by design, so it just sat idle. The supervisor was right that it had gone silent and started a replacement, but left the idle original sitting in its window to wake up later, so for most of your window two drivers ran one board, aimed helpers at each other's jobs, and couldn't message each other for forty minutes. The morning replacement was then cut off by a usage limit for three hours. Almost everything was diagnosed and fixed between 11:40 and 12:35, just after your window.
| Time | What happened |
|---|---|
| Fri 8:13pm | The first driver takes the board. |
| 8:49–10:17pm | The alive-check fails four drivers in a row; each is shut down and replaced while it is still typing, taking its helpers with it. |
| 10:34pm | Gera's session starts the night driver by hand. |
| 10:37pm | Sixty seconds later the alive-check fails it too — that's the fifth strike. The supervisor shelves the whole job and tries to escalate, but there is no one to tell. The driver keeps working anyway. |
| 10:49pm | A replacement is already running the same board. Two drivers, one board. |
| 11:32pm | The account hits its ceiling; helpers die. Account-switching swaps the driver onto a fresh account and it carries on. |
| Sat 3:20am | The board's finished count stops moving. It stays frozen for nine hours. |
| 5:46am | The night driver quietly stops. Not a crash and not the usage wall — it had written itself a rule to hold above 70% of the account, refused to keep going six times, and the safety gate gives up after six refusals by design. Nothing was left to start its next turn, so it sat idle. |
| 6:02am | The supervisor sees the silence and starts a morning replacement. Two drivers again. |
| 6:04–6:13am | The supervisor tries nine times to wake it and is told "it isn't running" while it sits there working. |
| ~9–11:50am | The morning driver is cut off by a usage limit for three hours. |
| 11:28am | Gera types "status of work?" — the idle night driver wakes back up. |
| 11:40am–12:35pm | The night's problems are diagnosed and most of the fixes ship. |
| 12:12–12:25pm | Gera answers 12 parked questions, some of them 13 hours old. |
Cost Four drivers killed mid-sentence, crews with them; a fifth declared dead a minute after it started and left running unsupervised for six hours.
Now A missing name on the list is no longer a death certificate — the check reads the agent's own activity before deciding.
Cost Twice it hit its limit, tried to raise an alarm, and found no address to send it to — so nobody was told the fleet was in trouble.
Now The check that misled it is fixed. Who receives its alarms, and rebuilding it automatically when a fix lands, are still open.
Cost Duplicated work, helpers aimed at each other's jobs, and forty minutes where two drivers couldn't reach each other.
Now Nothing yet stops a second driver from starting on a board that already has a live one.
Cost The board couldn't mark anything finished for nine hours — the tool to move a row up to "done" didn't exist.
Now That tool exists, and it refuses to close a row without a reason. But the honest audit found zero of the "finished-looking" rows were actually done, so the count is still 163 — that part is real work left, not a bug.
Cost Roughly nine checks quietly passed without running — a receipt written for a test that never ran, a broken guard waved through. A missing check looked exactly like a passing one.
Now The named ones are closed; a couple of "did it really run?" gaps remain.
Cost At the account ceiling, eight to ten helpers died in the same second; the driver then invented its own "stop early" rule, and the morning driver lost three hours to the limit.
Now The "never stop just because you hit the limit" rule is written down; nothing yet shares or meters the account across helpers.
Cost Three surfaces disagreed on how many questions needed a human — one said 6, another would show 7, the true answer was 0. A founder couldn't trust the number.
Now A single shared definition is written and shipped as a change, not yet merged and switched on.