A retro on the last 14 days of #alerts for Fede and Gera: what posted, who looked, why it keeps growing, and the fixes, each with its pull request.
Written Nov 4, 2026 · window Oct 21 00:00 UTC to Nov 4 14:50 UTC (Fede's "way too noisy" post) · all times UTC · Fede's #alerts thread · updated Nov 4 19:30 UTC: 9 fixes merged, 3 wait for Gera
Nothing in #alerts ever goes away. Every way an alert gets "handled" is one more post: a verdict in the thread, a "back to normal" reply, a page asking a person to look. And no verdict ever changes the check that fired, so a false alarm comes back on its next cycle and gets judged from scratch.
About a third of the channel (105 of 355 alerts) is our own agents and tools talking about their own work, and 87% of all thread replies are Smith. Only 19 of the 355 alerts got a reply or reaction from either of you.
Earlier fixes went in at whichever of about five posting doors the complaint came through, so the next flood came through another door. Nothing counts posts per day. The fixes for the false alarms we did find are written and approved but sat behind a hold label that nothing was reading; a fix for that is now open (agent-harness#1923).
It is like a smoke alarm that rings every time someone makes toast, where the only response anyone can give is to write a note and stick it on the wall. After two weeks the wall is covered in notes and the alarm still rings at toast.
| Kind | Posts | Fires | Nobody engaged | Zero replies | What is in it |
|---|---|---|---|---|---|
| Agent bookkeeping | 47 | 47 | 46 | 41 | "Waiting on a person" pages about agent stalls, Smith's daily ledger line, Smith's top-level verdict lines |
| Fleet tooling | 58 | 175 | 54 | 39 | reviewer-token pool, Trinity's self-update, the Opus metered-key cost page, mini cron jobs |
| Deploy and CI | 23 | 23 | 23 | 0 | "production deploy failed", release staleness, deploy freshness |
| Nightly tests | 14 | 14 | 12 | 0 | nightly maintenance and morpheus evals, nightly voice regression watch |
| Product | 213 | 264 | 201 | 48 | Clara, customer data, AWS alarms, Sentry |
| Total | 355 | 523 | 336 | 128 |
Not in the 355: Sentry also broadcast 16 thread replies into the channel feed (10 from the AppFolio work-order enrichment issue, 5 from the lead-outcome sweep, 1 other). They are replies, but they are part of what you wake up to.
Days are UTC days. The busiest day, Nov 3, was 46 posts and 191 thread replies. The quiet-looking first week fired more than it posted: Oct 24 was 21 posts but 63 fires, mostly the reviewer-token pool repeating.
Trinity counted the channel first, in Fede's thread. Lane r1 re-derived every number from a fresh read of the channel. Where they differ, it is almost always the predicate, named below.
| Claim | Who | Re-derived | Agree? | Why |
|---|---|---|---|---|
| 360 top-level posts Oct 21 to Nov 4 | Trinity | 360 / 355 | yes | 360 = every channel row in the rolling 14 days to her post (thread broadcasts and system rows included). Bot alerts only, from Oct 21 00:00 UTC to the cut: 355. |
| 1,228 thread replies | Trinity | 1,265 | close | 1,265 by reply time from Oct 21 00:00 UTC to the cut; 1,232 in the rolling 14 days. Her exact 1,228 could not be reproduced (within 3%). |
| 41 alerts answered "fix written, waiting on a person" | Trinity | 41 | exact | A Smith reply matching "held for (a person | review | Fede | your review)". |
| 130 (36%) got zero replies | Trinity | 128 | close | 128 bot alerts with zero replies. Her list includes Sentry broadcasts, which are replies; adding those gives 144. 130 not reproduced exactly. |
| 15 of 19 "production deploy failed" went green on their own | Trinity | 15 of 19 | exact | A "main green" reply in the alert's own thread. |
| Duplicate-guest-card lead page called "noise" 4 times | Trinity | 4 | yes | Lead-outcome posts whose verdict says noise. |
| About 6 noise verdicts were our own test traffic | Trinity | 4 to 11 | bracketed | 4 where the verdict says plainly "noise"; 11 counting "code fix (false alarm from a test building)" too. |
| Reviewer-token rotation, 16 zero-reply posts | Trinity | 16 / 34 | different cut | The rotation posts alone are 16; the whole reviewer-pool family is 34 posts, 26 with zero replies. |
| 359 alerts in 14 days | Smith | 359 | exact | Every channel row in the rolling 14 days to Fede's post. |
| Only 20 alerts (6%) got a reply from a person | Smith | 19 | yes | The 20th is a deleted message, dropped as a system row. |
| About 105 of 359 are about our own agents and tooling | Smith | 105 | yes | Agent bookkeeping plus fleet tooling. |
| Volume grew from about 10 a day to 46 on Nov 3 | Smith | 23 → 46 | half | 46 on Nov 3 matches. "About 10 a day" does not: the lowest day in the window is 14 and the first week averaged 20. |
| Alert source | Kind | Posts | Fires | Days it fired | Nobody engaged | Smith said false alarm | Fix held for a person |
|---|---|---|---|---|---|---|---|
| Reviewer-token pool | fleet | 34 | 151 | 11 | 33 | 1 | 0 |
| Sentry, one-off issues (28 different) | product | 28 | 28 | 9 | 26 | 7 | 4 |
| "Waiting on a person" owed-item pages | agent | 22 | 22 | 3 | 21 | 0 | 0 |
| Production deploy failed | deploy | 19 | 19 | 12 | 19 | 0 | 0 |
| Conversation grade red | product | 18 | 18 | 8 | 17 | 6 | 3 |
| AWS CloudWatch alarm | product | 16 | 54 | 10 | 14 | 2 | 2 |
| Contact match refused | product | 16 | 16 | 10 | 16 | 2 | 0 |
| Stale-escalation daily report | product | 15 | 15 | 15 | 15 | 1 | 3 |
| Smith's daily alert ledger | agent | 14 | 14 | 14 | 14 | 0 | 0 |
| Smith's top-level verdict lines | agent | 11 | 11 | 10 | 11 | 0 | 2 |
| Automation run failure | product | 10 | 10 | 8 | 10 | 2 | 4 |
| Nightly eval cron | nightly | 10 | 10 | 8 | 10 | 0 | 3 |
| Trinity's Opus metered-key cost page | fleet | 10 | 10 | 4 | 7 | 0 | 0 |
| Lead-outcome sweep | product | 9 | 9 | 7 | 9 | 4 | 0 |
| Trinity's auto-update | fleet | 6 | 6 | 1 | 6 | 0 | 0 |
The reviewer-token pool's single worst title fired 80 times. Under the Sentry one-off issues Smith wrote 204 thread replies, the most of any source.
One per mechanism. Each says what happens and which removal path is missing. Code read at propflowai main 8c2ac122cbe and agent-harness main 89e39b3cf, both Nov 4.
A verdict, a "back to normal", a "same error class" note, a page to a person: each is one more message. The original post is never edited or marked resolved. Only one emitter edits its own post in place when it recovers (the PMS-outage card), and it is the pattern the rest should copy.
Missing: an edit-in-place on recovery.
"Noise" is a sentence in a thread and nothing else. Smith's earlier verdicts on the same source are shown to the next run as background, never counted, so the 4th "noise" is decided like the 1st. The repeat key is built from the records named in the alert, so a check that names a different lead each time looks brand new every time. That is how the duplicate-guest-card lead page was called noise four times. The design doc named this hole as a follow-up; it was never built.
Missing: a path from "judged false twice" to a change to the check.
The check that knows a test building, a sandbox company or a robot phone works. But each poster has to ask it. 40 files still post to #alerts directly around the one shared alert door, and up to 14 of them never mention a test check. The door itself only recognises a test contact when it is an email, so a test phone at a real building pages. Sentry posts straight into the channel and has no idea what a test is.
Missing: one gate every post passes, phones included.
Fede's Sep 15 ruling ("Please stop the noise") was wired into one path: a bot re-fire of an answered alert. The claim note, the red-main owner note, the "same error class" note, the record join (two messages each) and the "came back after a fix" note all still post a threaded line, and so does every verdict that is not a silent OK. Smith also forgets an alert after 48 hours, so an alert that stays red gets a fresh investigation and a fresh verdict every two days. Since Oct 27 every #alerts post wakes Smith, FYIs included.
Missing: the reaction rule applied once, for every path.
A repeat on the same UTC day becomes a count reaction. A condition still broken the next day posts a fresh top-level alert: "production deploy failed" posted on 12 days, the stale-escalation report on 15, Smith's ledger on 14. Midnight UTC is 5 pm in Denver since Nov 1 (6 pm before). Trinity's posts skip the fold altogether: 6 auto-update posts on one day, the Opus cost page 10 times in 4 days.
Missing: a fold keyed on the open problem, not the calendar day.
Muting an alert only stops Smith from investigating it; the emitter still posts. Three muted alerts posted in the window. And Smith's three volume tools (the DM bouncer, the burst gate, the repeat collapse) all skip alert posts on purpose, on the rule "a lost alert is worse than a noisy one", so none of them can ever thin #alerts.
Missing: a mute that reaches the emitter, and any cap on posts per day.
The "a stall pages" rule (agentflow ADR-0080) picked #alerts as the place for every owed-item page: prompts with no decision row, expired drives, internal handoffs. Nothing separated "a customer is waiting" from "an agent lane stalled", and each page tags Gera.
Missing: a second address for pages that are about our agents.
Smith's own fix-PR rule (Oct 20, every fix branch since Oct 25) opens every alert-fix PR with hold-for-review, and since Nov 1 the cross-review also adds it when a review is requested (cross_review.py, _hold_at_request). The only release is the other seat labelling the fix root-cause and low or medium risk, which exists only since Nov 1 and runs only inside a PR drive. Drives expire after 36 hours. Four finished fixes, each with a 🟢 as its latest bot verdict, have sat since before that release existed:
This hold is not Fede's Oct 22 rule. Fede's own scope of that rule, in #engineering-bridge at 19:04 UTC that day:
"I meant the individual features getting deleted, fixed or turned on. Not the whole initiative"
Nothing in his words names #alerts or alert-fix PRs. But Smith's rule store keeps both readings marked binding: the broad one ("any change to a gate, a decision or a behavior", Smith's own paraphrase, which Smith withdrew 86 minutes later) sits where PR work reads it, and Fede's narrow one sits in a different section.
Missing: a releaser that outlives the drive, and one copy of Fede's rule.
| Earlier fix | Date | Still in force? | Why the noise came back |
|---|---|---|---|
| Fede: "Please stop the noise." A bot repeat gets a reaction, no reply (agent-harness#566) | Sep 15 | yes | Never covered the emitter's own repeat post, only Smith's reply to it, and only within 48 hours. Every later flood was an emitter re-posting. |
| September noise audit: no @-mentions in #alerts, one bot per alert, reports leave | Sep 12 | partly | The rule stands. Since Nov 1 the merge drive's escalation line tags both founders; where it posts was not read. |
| Fede's noise diet: CI, deploy and fleet posts leave #alerts | Oct 14 | no | Reversed two days later by Gera's Oct 16 ruling: "broken things (deploy failed, CI/main red, bake alerts, security/token alerts, Sentry) go to #alerts. Overrides Fede's 2026-10-14 call." The guard that pinned them out was flipped to pin them in. |
| A broken job posts once per failure streak, not once per run | Oct 14 | opt-in | Never covered by default: only 7 of 58 files that call the alert door pass a streak key. |
| #alerts has one door; a shrink-only list of files that already posted directly | Oct 15 | yes | Bandage on the mechanism: it stops new direct posters but never moved old ones. The list went 36 → 42 → 40 in 20 days; one file was moved onto the door. |
| Gera: machine posts go to his DM; a bouncer at Smith's Slack door | Oct 14-16 | yes | Never covered #alerts by construction (it skips alert posts), and it moved noise rather than cutting it: Gera's DM took 301 automated posts in 22 hours. |
| Burst gate: the 6th top-level post in 2 minutes is folded | Oct 27 | yes | Never covered #alerts: alert posts are carved out on purpose. |
| Reviewer-token pool stops reposting on every unchanged re-fire | Oct 18 | half | Half landed: the stable key merged; the count fix (dotclaude#277) is still open, and the checker's saved position has not updated since Oct 10, so it re-reports what it already paged. |
| Robot alert dedupe; tour-calendar flood | Oct 14, Oct 24 | merged | Bandage per emitter: grouping added to one emitter at a time, not to the door. (The tour flood hit a customer channel, not #alerts.) |
| Reply-not-sent alert reroute (propflowai#23757) | Oct 30 | yes | The good fix: Trinity sent the first try back as "correct behaviour, wrong door", and the redo moved the file onto the one door. |
| The Smith noise-diet board | Oct 14 on | open | Drifted: 96 rows, about 16-20 of them about posts or channels; the rest became merge-gate and lint work. The two rows about volume today are still open. |
The pattern. Every fix went in at the door of the person who complained, and #alerts has at least five doors: the shared alert door, the 40 files still posting around it, CI workflow posters, Sentry's own Slack rules, and Smith's alert posts. Two founder rulings two days apart pointed the same family of posts in opposite directions, and the code obeyed both in turn, so posts moved between #alerts, #agent-smith and Gera's DM without the total going down. The guards that did ship stop new mistakes but never shrink old ones, and nothing anywhere counts how many posts #alerts takes in a day.
How these merge (settled Nov 4, on Fede's own Oct 22 scope): alert fixes follow the normal reviewed flow and merge on a clean bot review and green checks. Three open PRs wait for Gera (decision b93811878): agent-harness#1923, agent-harness#1924 and propflowai#25205. propflowai#25155, the deploy-alarm timing fix, merged on its own before Gera approved it, because the driver did not hold it. Gera has been asked to keep it or revert it.
Posts removed are lane r1's 14-day counts, one source label per post, so the rows do not overlap. They are what each fix could remove. Where a fix's own PR measured its effect, that number is shown instead; hover a number for its source. Some rows as built move less than r1's estimate, because some pages stay on purpose. Lane f1 measured its own effect on #alerts top-level posts from Nov 3 08:00 UTC to Nov 4 14:00 UTC: 41 posts, of which its PRs move 7 (5 pages plus 2 roll-ups).
| Fix | Repo | Posts removed (14 days) | Turns a check on or off? | PR |
|---|---|---|---|---|
| "Waiting on a person" pages that are Gera's alone go to his DM, untagged. Two kinds stay in #alerts on purpose: Fede's own stall pages (Fede, Oct 6: never a DM to him, so moving them would delete his notice), and Trinity's pages (her bot cannot post into Gera's DM with Smith) | agent-harness, agentflow-relay | part of 22 | No, routing only | agent-harness#1921 merged agentflow-relay#493 merged |
| Smith's alert-ledger roll-ups that do not thread under an alert go to Gera's DM, not #alerts top-level | agent-harness | 14 | No, routing only | agent-harness#1922 merged |
| Smith's top-level verdict lines go into a thread. A failed sync-canary run now tells the decider which thread to answer in, so its verdict threads under the queue thread instead of posting top-level | dotclaude, agent-harness | 5 of 11 | No, routing only | dotclaude#363 merged agent-harness#1925 merged follow-up The other 6 lines come from other jobs and are not threaded yet. |
| Routine reviewer-token pool notices (rotation warnings, running-low) go to Gera's DM. The possible-security writer-check page and the red "no account answers, reviews will fail" outage line stay in #alerts, per Gera's Oct 16 ruling. The routine 🟠 rotation warning fired 17 times in 11 days with nobody acting on it, so recalibrating that check is the real follow-up | dotclaude, agentflow-relay | 22 of 34 | No, routing only | dotclaude#362 merged agentflow-relay#494 merged |
| "Production deploy failed" waits 45 minutes and pages only if production is still behind. Of the 19 posts in the window, 12 were the borrowed build machine taken away mid-deploy, 6 were failed Vercel builds, and 1 was a run of 5 failures. Separately, the production build's main memory driver (a build-cache write nothing used) is removed; the next production build's printed memory peak is the proof | propflowai | 17 of 19 | No: changes when it pages, not whether | propflowai#25155 merged before Gera approved it; he has been asked to keep or revert it propflowai#25168 merged (build memory, part 1) propflowai#25175 merged (build memory, part 2) |
| The stale-hold report stays in #alerts but posts only when the list of held items changes | propflowai | 5 of 15 | No, posts on change | propflowai#25161 merged |
| Retire or quiet the nightly test suites (nightly evals, nightly voice regression watch) | not traced in this pass | 14 | Yes, deletes checks | needs a founder's go Fede asked on Nov 1 and Nov 3 and Gera asked "why not just remove them"; the go still needed is on exactly which suites. What each suite is for and what it becomes: nightly suites page. |
| Grade-red pages are verified before they page | propflowai | 6 | No: fixes the check's condition | propflowai#21822 written, still held (since Oct 22) |
| Trinity's Opus cost page posts once a day, not every hour | Trinity (Fede's machine) | 6 | No | not opened Trinity's code lives on Fede's machine and could not be read from here. |
| Trinity's own update failures go to her owner, not #alerts | Trinity (Fede's machine) | 6 | No | not opened Same reason. |
| The Bench's deliberate read-only refusals stop paging Sentry | propflowai | 5 | No: marks test traffic as test | propflowai#22284 written, still held (since Oct 24) |
| A second delivered copy of an answered lead email is no longer read as a dropped lead (the duplicate-guest-card page Smith judged noise four times) | propflowai | 4 (+5) | No: fixes the check's condition | propflowai#25158 merged |
| Total (r1's estimate) | 152 | |||
| Not in the total: Sentry's channel broadcasts on two ongoing issues (15 lines in the window). Fix by resolving or archiving the issues or turning off the broadcast. | ||||
These fix the root causes above. Their posts overlap the rows above, so they are not added to the 152.
| Fix | Root cause | Repo | Expected effect | Turns a check on or off? | PR |
|---|---|---|---|---|---|
| Test traffic stops paging: the alert door recognises test phones, and the escalation guard-rejection alert checks the building's test status | 3 | propflowai | Trinity counted about 6 test-traffic noise verdicts in 14 days; r1 brackets it 4 to 11 | No: a correctness rule, no switch | propflowai#25151 merged |
| Move the direct posters onto the one door, starting with the up-to-14 that never check for tests | 3 | propflowai | not measured | No | not opened The next step after propflowai#25151. |
| Every Smith note under a bot alert becomes a reaction, on all six paths; a "noise" verdict becomes a reaction too; a mute stops the post; and Smith's Oct 27 "joined #thread" line is dropped. Open question in the same decision: the lines that name an owner | 4, 6 | agent-harness | Mostly thread replies, not posts. Of Smith's 1,092 replies under these alerts, 194 are verdicts and 53 are re-fire notes; the exact cut was not measured. Muted alerts that still posted: 3 in the window | No: honours a mute someone already set | agent-harness#1924 waits for Gera |
| Two "noise" verdicts on one source: mute it for a set time and open one fix task on the check itself | 2 | agent-harness | Every repeat of an alarm already judged false; not measured | Yes, mutes a check for a time | needs a founder's go A mute alone is a bandage under the Oct 19 "root fix over bandage" ruling; pairing it with an expiry and a mandatory fix task is the proposal. |
| Held fix PRs get a release that outlives the drive: a clear risk label lifts the hold whenever it lands; the ADR's attribution window matches the code (15 s) | 8 | agent-harness, agentflow-relay | Unblocks held fixes such as #21822 and #22284; removes the founder and drive-expired pages about them | No | agent-harness#1923 waits for Gera agentflow-relay#495 open (ADR-0078 amendment) |
| The merge hold re-reads the "dangerous diff" check when it finishes, instead of treating a stale "not passed" as final. Today it held 4 PRs at once | 8 | propflowai | Stops fix PRs being held by a check that already passed | No | propflowai#25205 waits for Gera |
| A repeat edits the first post instead of posting fresh each UTC day | 1, 5 | propflowai, dotclaude | not measured; "production deploy failed" alone posted on 12 days | No | not opened dotclaude#277 (open since Oct 18) covers one emitter only. |
| A daily count of #alerts posts by source, against a budget you can see | 6 | not chosen | Makes a new flood visible the day it starts | No | not opened Needs a home and a number. |
Three read-only research lanes of task alerts-noise-retro, Nov 4: r1, a fresh 14-day census of #alerts (every post, reply and reaction, with the predicate beside each number and positive controls on each count); r2, the posting and dedupe code read at propflowai 8c2ac122cbe and agent-harness 89e39b3cf; r3, the earlier rulings and fix PRs, with Fede's Oct 22 words read from #engineering-bridge. Trinity's and Smith's first counts are from Fede's #alerts thread. No customer names, phones or addresses are on this page; test companies are named where the finding is about them.