Sixty days of alerts, incidents and misses, measured. One severity rule, three ways to reach a person, and a recommendation.
2026-10-14 · Aug 14 to Oct 13, 2026 · Slack, Sentry, CloudWatch, the robot, the app, Fede's inbox · times are Mountain · Proposed — pending Fede
New page, not a section of the Oct 12 bot-noise page: that one is about how the bots write, this one about whether a person is reached. The Oct 14 RCA covers the write queue; both link here.
TL;DRNoise dietAgent on-callYesterdayNumbersEvery pathSilent failures60-day ledgerDesignOptionsPR chainAsksAppendix
Goal (Fede, Oct 14): 80% fewer posts in #alerts, proven within 25 hours. Two baselines, both counted with the same script (count-alerts.py and the raw pull sit next to this page). The 14-day rate was 65 a day (914 posts), but 47% of those families had already gone quiet before Oct 14 (the CI storm, Willows bounces, held-for-review cards). The honest live rate is the 44.6 hours before the first merge: 59 posts, 31.8 a day. 80% off the live rate is 6.4 a day; 80% off the 14-day rate is 13 a day.
| What posts to #alerts | 14 d | Live, per day | Change | Status |
|---|---|---|---|---|
| CI, deploys, merge guard, fleet | 254 | 12.9 | Moves to the bot channel #agent-smith | Merged Oct 14, 3:55 PM MT |
| AWS cost anomalies and budgets | 123 | 3.2 | Dropped from #alerts; the AWS side (budgets, anomaly bar) is a separate ask below | Merged Oct 14, 3:40 PM MT |
| CI fleet vCPU lines (Mac poller) | 82 | 0 | Moves to #agent-smith (had been silent since Oct 1; no forward effect) | Done Oct 14, 2:45 PM MT |
| Test properties (Willows, Fairhaven) | 163 | 1.1 | Never posts; logged only | Merged Oct 14, 3:20 PM MT |
| A broken job re-posting every run (bake alerts, budget exhausted, contact coverage, PM reminder) | ~60 | 2.7 | Once per failure streak or state change, once on recovery | Merged Oct 14, 3:10 to 3:58 PM MT |
| Informational records and stale "conversation needs a look" | ~85 | 1.6 | Records go to #agent-smith; sweep looks at the last 2 days, once per conversation | Merged Oct 14, 2:55 PM MT |
| Dead and non-prod CloudWatch alarms (51 of 106) | 20+ | 0 | Deleted | Done Oct 14, 2:30 PM MT |
| Held-for-review cards | 52 | 0 | All test cards; stopped Oct 13 (#19142). The earlier ask is withdrawn. | Done before today |
| Sentry (nine rules post to #alerts; the main one is "new high-priority issue, first seen, production") | 85 | 8.1 | Judged by content with a second reviewer: of 18 posts, 3 were incidents a person acts on, 5 tickets, 3 test or false alarms, 4 repeats of a job failure that already alerts, the rest restate one open incident. App fix (no switch, PR in progress): tag the property, demote test properties, duplicate job failures and downstream refusals to warning. Sentry side is the ask below. | Done Oct 15, 9:40 AM MT on Fede's go: "new high-priority issue" and "scheduled job failed" now post to #agent-smith, the every-15-minute renewal rule is disabled (not deleted), and a new rule pages #alerts only for errors tagged customer-impact (the app tags them; PR in flight). App fix merged Oct 14, 5:40 PM MT. |
| Smith's own posts (deploy-freshness banners, verdict ledger) and the standup recap "degraded" line | ~40 | 4.3 | Handed to Smith on the bridge: out of #alerts, or once a day | Handed off Oct 14 |
| Clara at a customer: phone hand-off broken (Western Slope catch-all), conversation needs a look, approved text not sent | ~75 | 4.8 | Stays. The hand-off alert is a real gap but posted 4 times for one gap; it now posts once per open gap per company (merged Oct 14, 5:25 PM MT; 9 posts become 1). | Keep, once per gap |
Where it lands (second reviewer's ladder, Oct 14 10 PM MT):
| Stage | Per day | Off the live rate |
|---|---|---|
| 14-day count (frozen) | 65 to 67 | |
| Live before the first merge | 32.8 | |
| Today's nine merges, replayed on the live window | 20 to 23 | 30 to 38% |
| + Second-pass fixes: streak re-arm and batching, hand-off once per gap, Sentry app fix, test people (all merged Oct 14 evening); kinds guard merged 6:30 PM MT | 16 to 18 | about 50% |
| + Smith's posts and the recap line out of #alerts (handed to Smith) | 9 to 11 | about 70% |
| + Sentry rules retargeted (Fede's go) | about 6 to 8 | 75 to 82% |
| 80% target | 6.6 or fewer | 80% |
It lands on the line with no margin, and only if every step ships. What remains then is real: customer-impact Sentry (3), conversation needs a look (4 to 5), hand-off (1), first posts of new failure streaks (2 to 3). Daily totals in the baseline ran from 12 to 197, so the 24-hour count on Oct 15 confirms the direction; the replay of the live window with the same classifier is the proof.
Proof, Oct 15, 3:30 PM MT (23 hours since the last Oct 14 fix deployed): 20 posts = 20.9 a day. Goal missed.
| Rate | Per day | Cut |
|---|---|---|
| Frozen 14-day count (Sep 30 to Oct 13) | 65 to 67 | |
| Honest live rate before the first merge (Oct 13 to 14, second reviewer) | 32.8 | |
| After, 23 h (Oct 14 4:30 PM to Oct 15 3:30 PM MT) | 20.9 | 68% off the frozen count, 36% off the honest rate |
| Same window replayed with today's four merges and the Sentry move (all landed 9 AM to 1:40 PM MT, after most of the window) | about 8 | about 75% off the honest rate |
| + Smith's ledger line out (handed to Smith) | about 7 | about 78% |
| Target | 6.6 or fewer | 80% |
The 20 posts, by family: 5 nightly data-integrity reports (fixed this morning, to the bot channel); 1 test-property AI fallback (fixed this morning); 9 Sentry, of which 2 were real (a life-safety page not sent; the JP&Co robot cold) and 7 were job-failed duplicates, downstream refusals of the JP&Co lockout and tickets (Sentry rules moved at 9:40 AM MT; the two real ones now reach #alerts through the customer-impact rule and the robot-cold page); 2 capture-drift findings and 1 guest-card outbox failure, real and kept; 1 Western Slope "lead with no first reply" page that was a false alarm (a second application card folded into an existing lead; fix in flight from another session, #20090); 1 co-signer roster line, the opening post of its streak; 1 Smith ledger line. The next 24 hours are the first with every change live; the count on Oct 16 at 3:30 PM MT is the number that stands.
Interim reading, Oct 15, 8 AM MT (first night with every fix live; all six follow-ups reached production by 7:33 PM MT Oct 14):
| Window | Posts | Per day | Same hours in the 14-day baseline | Change |
|---|---|---|---|---|
| Overnight, 10 PM to 7 AM MT (9 h, all post-deploy) | 15 | 40 | 22.9 per night (overnight is the dense hour: nightly checks run 2 to 3 AM) | 34% fewer |
| Trailing 24 h (includes the afternoon before the fixes deployed) | 34 | 34 | 65 to 67 | about 48% fewer |
What the 15 overnight posts were, and what happens to each: 5 nightly data-integrity reports from the spine reconcile cron, which never finished before a cost fix on Oct 14 and now posts its findings (PR in flight this morning: they go to #agent-smith, Smith's lane); 1 AI-fallback alert about the Willows test property, a new poster from Oct 14 that skipped the test-property rule (PR in flight); 4 capture-drift posts for 2 real findings, each doubled by Sentry's "scheduled job failed" rule, which matches on a tag, not on level, so the app-side demotion cannot stop it (the Sentry ask above); 1 co-signer roster line, the opening post of its newly fingerprinted streak (quiet tomorrow if the same row); 1 Smith ledger line (handed to Smith); 3 Sentry overnight, of which the life-safety page not sent is a real incident, the JP&Co cold login and paused inbox are one incident in two posts, and the work-order enrichment partial sweep is a ticket. With the two PRs merged and the Sentry rule moved, the same night would have been 5 posts, about 13 a day on the dense hours and under 7 on a normal day. The 3:30 PM MT count is the proof; this one is the direction.
Sentry ask (Fede's call; a Sentry UI change): point the "new high-priority issue" and "scheduled job failed" rules at #agent-smith, delete the every-15-minutes renewal-prepare rule (it called the AppFolio key outage a renewal failure), and add one rule that pages #alerts only for errors tagged customer-impact. Recommend yes. Same-day asks still open: unhook the AWS budget and anomaly subscriptions from Slack and raise the two daily budgets to $150; weekend tour check-in fix (one line) or remove.
Fede's ask (Oct 14 night): "I want to be tagged if it's a real incident, production down, loud but very rare. Small bugs should be picked up and fixed by agents. And I don't want agents piling up pull requests that never get merged." Three read-only inspections, no code: 90 days of #alerts against every real incident, every PR Smith has opened, and what other teams publish. Full reports sit beside this page in the scratchpad of the authoring session; the tables here are the parts that decide things.
| Jul 16 to Oct 14 | |
|---|---|
| Posts in #alerts | 3,442 (265 a week) |
| Posts that pointed at a real incident | about 3% |
| Real incidents (customer-facing wrong or missing behavior) | 157, about 12 a week |
| Outages (production down or a customer path dead) | 33, about 2.5 a week |
| Outages where Fede was tagged | 5 fully, 2 partly; none within the first hour; the earliest tag came 4 hours in |
| Time from first signal to a person knowing (18 outages with a recorded time) | median about 12 hours |
| Outages found by an alert / a session doing other work / a customer / a person / not recorded | 5 / 10 / 3 / 3 / 12 |
| Outages with no signal anywhere | 11, mostly transfers and leads that got no reply |
Where it broke: the AppFolio robot and sync (12 outages), phone and voice (10), lead and reply email (6). Sentry saw six outages first, with thousands of errors, and paged nobody because the only rule fires on brand-new issues. The three night outages ran longest because nobody reads alerts before 7 AM.
| When | What broke | Found by | Signal to a human | To fix | Fede tagged |
|---|---|---|---|---|---|
| Aug 14 to 15, night | Camellia overnight writers got only the canned fallback reply | session (posted in #alerts overnight) | ~10 h | ~12.5 h | no |
| Aug 15 | Camellia team notices bounced, one signature email lost | session | n/s | same night | no |
| Aug 22 to 25 | Camellia website-form leads got no reply | session | ~3 days | ~13 h after first casualty | no |
| Aug 22 to Sep 1 | 7 of 19 transfers failed; 5 callers dropped, nobody told | customer | ~10 days | Willows only | no |
| Aug 26 | No property manager at any real property could be paged | n/s | n/s | 1.9 h | no |
| Sep 5 to 6 | Western Slope prototype line gave callers nothing | n/s | n/s | 9 h | no |
| Sep 9 | Robot logins locked at JP&Co and Situs | alert | ~0 (codes hit Fede's phone) | 1.9 h | indirectly |
| Sep 9 to 13 | AppFolio rate limit storm; Western Slope re-import missed 75 units | n/s | ~48 h | ~3.5 days | yes, ~48 h in |
| Sep 9 to 11 | Texts to the Western Slope line went nowhere | n/s | n/s | ~1.5 days | no |
| Sep 10 to 12 | Robot logins broken at all three AppFolio clients | alert (Sentry in minutes) | ~15 h | 17 to 35 h | yes, ~15 h in |
| Sep 12 to 30+ | Sync jobs off; stale lease end dates at every customer | n/s | ~3 days | 18+ days | yes |
| Sep 14 | Inbound texts delayed up to 47 min, one lost | session (cost audit) | ~3 days | n/s | no |
| Sep 15 to 20 | Western Slope prospect replies not read | n/s | n/s | ~5 days | no |
| Sep 16 | 19 calls failed, 0 got through | person (test calls) | ~80 min after deploy | 1.8 h | no |
| Sep 18 to 25 | Western Slope caller dropped, contractor refused | Fede | Fede himself | ~7 days | found it |
| Sep 18 to 22 | Western Slope callers dropped on transfer | session | ~4 days | ~4 days | no |
| Sep 19 to 23 | Weekend Western Slope leads got no first email | session (logs) | ~1.5 to 2.5 days | Sep 21 | no |
| Sep 21 | Western Slope callers handed to staff were dropped | customer | 0 (their email) | 13 min | no |
| Sep 24 | One property's guest cards unsynced | Sentry only | nobody | 2.5 h, cleared itself | no |
| Sep 25 | 3 real calls lost their follow-up; first report said nobody affected | n/s | n/s | ~4 h | no |
| Sep 26 | Cache cap hit; robot writes stalled for every client | Sentry only | ~5.7 h (quiet hours) | 11.8 h | yes |
| Sep 26 to 28 | Camellia concession watcher failed; test texts unanswered | session (weekend audit) | ~18 h | ~32 h | no |
| Sep 29 | Situs rate-limited within minutes of connecting | n/s | n/s | ~12 h | no |
| Sep 30 to Oct 1 | Western Slope Zillow leads got no reply | n/s | ~9 h | ~28 h | partly |
| Oct 7 | Western Slope robot work failed on the first self-guided tour night | session (retro) | ~3.9 h | ~5 h | yes, 3.9 h in |
| to Oct 8 | Connected Gmail mailboxes could stop refreshing | n/s | n/s | n/s | no |
| Oct 8 | Western Slope user locked out; staff sign-in codes sent from Camellia's line | customer | 0 (their email) | 1.8 h | partly |
| Oct 12 to 14 | Western Slope robot paused each morning | n/s | n/s | ~2 days | no |
| Oct 12 to 13 | JP&Co new guest cards not read | session | n/s | ~23 h | asked for a grant, not paged |
| Oct 13, 02:55 | 3 calls had no audio (vendor) | session (overnight) | n/s | 40 min | no |
| Oct 13 to 14 | Dead robot key; 11 guest-card writes held up to 20.5 h | alert (#alerts in 37 min) | 37 min post, read next morning | ~20 h | no |
| Apr to Sep 2 | Yale feed told prospects nothing was available | person (go-live review) | months | Sep 2 | no |
| Apr to Oct 7 | Website demo requests lost for six months | n/s | months | Oct 7 | no |
n/s = not stated in the record. The 124 non-outage defects are in the full report.
If Fede were tagged on: every #alerts post, 265 pages a week, 97% noise. Only posts marked customer-impact today, 31 a week, still about 9 in 10 noise. One page per outage, the target: 2 to 3 a week, zero noise.
| Since Jul 9 | |
|---|---|
| PRs opened | 662 (561 alert fixes, 96 Slack-asked jobs, 5 other) |
| Merged | 541; median 25 minutes to merge, 9 in 10 within a day |
| Closed unmerged | 120: 82 duplicates (another Smith run or a person landed the same fix first), 26 abandoned and swept, 7 rejected by an owner, 5 other. 84 closed within 2 hours. |
| Open now / stuck on a red check or unanswered review | 1 / 0 |
| Reverted | 1 |
| Same alarm patched again within 7 days | about 20 chains, mostly alarm exceptions widened 2 to 4 times |
| Named a plan / had real testing evidence | 18 of 183 / 23 of 173 |
| "Fix the red main build" PRs | 23% of everything; 46% thrown away since the one-driver rule (22% before) |
| Review rounds spent on yellow suggestions | 203 extra rounds on 111 PRs; its instructions tell it to fix every suggestion |
Verdict. Smith does not pile up PRs; it ships in minutes. It does not have the design context: its instructions hand it the alert and "follow the repo guide", nothing points at the planning docs, Gera's portfolio design, the testing ladder or Fede's rulings. Of 12 PRs read in depth: 4 aligned, 5 shallow but harmless, 3 wrong, and all 3 wrong ones made an alert go quiet: it turned off the page for a blocked AppFolio login while the replacement alarm was off in production (Western Slope's robot then sat locked out 21 hours, Gera corrected it); it narrowed the autonomous-mode guard 43 seconds after a person opened the same fix (reverted in 40 minutes); it loosened a code rule a session had said was the owner's call, and merged first (still in place). Its "high risk waits for a human" promise is false: since Sep 12 its merge loop merges anything green with no risk check, and identity and Clara-behavior changes went in that way. Tonight's two PRs (renewal page crash, slow rent roll) had correct diagnoses and correct fixes; one stalled unarmed, one broke its own test and self-held.
No company publishes an audited setup where agents run on-call end to end; vendors describe it. The consistent practices: agents investigate and fix in narrow, reversible lanes; severity is defined by customer impact in numbers, never labels; the human page is rare and arrives with a summary; duplicates merge by key and flappy alerts pause. The closest precedent for self-merging agent fixes is Vercel, which reports 58% of PRs in its main repo merging with no human review and merge time falling from 29 to 11 hours. Sentry's own fixer opens PRs and nudges one reviewer, never twice. Sources in the research report.
error or failure
├─ customer affected, or production down?
│ yes ──▶ #alerts: one post, @Fede, stays until "got it" (2 to 3 a week)
└─ no ──▶ #agent-smith: one post per issue, nobody tagged
Smith works in the THREAD: reproduce → fix PR → bot review + CI → merge → post edited to ✅
cannot fix in a few hours, or customers turn out to be hit ──▶ promoted to #alerts, @Fede
morning digest in #updates-fede: fixed on its own / ticketed / escalated
The five checks that put the real incidents in #alerts (each pages once per outage, and pages if the check itself stops running):
| # | Check | Pages when | Outages it would have caught |
|---|---|---|---|
| 1 | Robot and sync heartbeat per connected account, paused ones included | no successful AppFolio cycle for 30 min, or writes held over 60 min | 11 |
| 2 | Inbound reply clock | a real lead or message has no Clara reply after 15 min, or a mailbox is unread for 30 min | 9 |
| 3 | Live call health | 3 or more failed, silent or under-10-second calls in 15 min, or a transfer with no pickup and no message | 8 |
| 4 | Staff notice delivery | a team notice bounces or is refused, or a property has nobody pageable | 2 |
| 5 | Error-rate jump on a customer path | a Sentry issue on the robot, reply or voice path passes 50 events in 10 min, once per issue | earlier warning on 6 |
Checks 1 to 4 cover 30 of the 33 outages. Checks 1 and 2 are the earlier audit's Option B; check 3 is new.
Smith policy changes (agent-smith repo, Gera's lane, sent over the bridge): quieting an alarm or loosening a check needs proof the replacement signal fires today, else it becomes a decision; one alert claims one PR across both machines and both sources; load the area's planning doc, the portfolio design and the testing ladder, and cite the plan in the PR; make the risk tier real (identity, AppFolio writes, Clara behavior get a real-data replay before merge) or delete the promise; suggestions become follow-up issues, only a red earns a push; a green, approved PR is armed to merge, never left for a person.
Built Oct 15 (Fede's go, after the JP&Co robot lockout): check 1 is live: an AppFolio robot cold for 15 minutes at a customer puts one post in #alerts with Fede tagged, once per outage per account, with a recovery line when it warms up (merged 1:40 PM MT). Customer-impact errors (life-safety page not sent, renewal notice failed, tour booking failed, staff notice undeliverable) now carry the tag Sentry's new #alerts rule pages on (merged 1:20 PM MT). The robot-side bug, 502 pages counted as failed logins, is Gera's lane, handed to Smith on the bridge. Known gap, filed as #20080: no daily "still down" reminder on a multi-day outage. Check 2 is live (merged 5:30 PM MT, #20151): every 5 minutes, at live companies only, a text or email where the person wrote last and nobody (Clara or staff) has replied for 15 minutes, excluding messages parked in the held-for-review queue, puts one post in #alerts with Fede tagged, once per company episode, with a recovery line. The mailbox-unread half of check 2 and checks 3 to 5 wait for the Oct 16 noise count.
The alerting worked as built, and that is the problem. Sentry fired, the triage bot named the dead key 45 minutes in, and then every path ended in a thread of a muted channel. Twenty hours passed. Gera and Fede were each in that channel during the outage, reading other threads.
| Path | Fired? | Where it died |
|---|---|---|
| Sentry issue alert → #alerts | Yes, at 2:03 PM (37 min), from another route's crash | A muted channel; the robot's own warnings never post |
| Smith triage verdict | Yes, 2:11 PM, correct | A thread reply and a Decisions-page block; no mention, no page |
| Trinity flag → #agent-trinity @Fede | No | Smith's hand-off step looks for the flag script on its own Mac, where it does not exist; Trinity stopped reading #alerts on Oct 12 at 6:34 PM |
| Robot's own alerts (login budget, breaker) | Yes, to Sentry | Counted a provider 401 as a sign-in failure; 24 retries burned the daily cap (fixed on the robot's main branch Oct 14, 10:44 AM) |
| CloudWatch alarms (106) | No | Nothing measures "robot cycles succeeded"; zero transitions in the window |
| Robot heartbeat (every 2 min) | No | Reports the process is alive, not that it can sign in |
| Held-write queue | No | Holds silently; the 15-minute sweep has no "oldest held write" alarm |
| Email, SMS, phone, push to a person | No | No such path exists |
The test for each source is what the post is about: must a person act on it, and could they? Reply counts are shown only to prove the channel is muted; they are not the test.
| Source | Posts | Share | Human replies (proof the channel is muted, not the test) | Belongs in #alerts? |
|---|---|---|---|---|
| CI, deploys, review bot (red main, harness red, deploy stuck, fleet capacity, review verdict waits) | 456 | 44% | 0 | No. Engineering plumbing; the bots own it. |
| Test properties (Willows, Fairhaven): bounced emails, held lead emails, renewal fixtures | 163 | 16% | 1 | No. Test data should never alert. |
| Clara at a customer (identity refusals, hand-off broken, stuck tours, held texts, outreach stalled) | 148 | 14% | 2 | Yes, batched: one post per problem, not per event. |
| AWS cost anomalies and budgets | 118 | 11% | 1 | No. A daily line in the cost report. |
| Sentry, customer-facing (login breaker, stuck texts, outbox, tour booking) | 39 | 4% | 1 | Yes, if it says which customer and what Clara cannot do. |
| Sentry, everything else | 38 | 4% | 1 | No. Sentry is the inbox for those. |
| Nightly evals, canaries, prompt drift | 27 | 3% | 0 | No. A morning digest. |
| CloudWatch alarms (one Mac mini heartbeat, 18 stage-replicator repeats) | 24 | 2% | 0 | Only the ones that mean Clara stopped. |
| Ledgers, reports, held-for-review queue | 14 | 1% | 2 | Reports no; the review queue has its own page. |
| Total | 1,027 | 100% | 8 (0.8%) | About 20% would survive the diet. |
Sixty days: 2,843 posts, 1.9% touched by a person; Smith attached a verdict to 19% of posts. When a person did reply, the median wait was 13 minutes.
| Measure | 14 days | 60 days |
|---|---|---|
| Posts | 164 (12 a day) | 388 (6.5 a day) |
| Posts that @mention Fede | 41 | 151 |
| Posts Fede replied to | 22 (13%) | 84 (21%) |
| Median wait for Fede's reply, when he replied | 52 min | 13 min |
| "A question has been waiting too long at <home>" (a Clara lead alert, posted by Smith, not a block) | 64 (39%) | 64 |
| Posts about the dead robot key, Oct 13 to 14 | 0 |
The Trinity flag is the only path built to wake a person; a session has to choose to raise it, and on Oct 13 none did. Two in five posts are a lead alert that belongs in #alerts.
| Issue family (60 days) | Events | Customer | What it means |
|---|---|---|---|
| AppFolio account budget exhausted | 25,311 | all | The sync's own rate guard, firing as an error. |
| AppFolio sync: phone coverage 0% (three property issues) | 24,838 | Western Slope | A data-quality warning repeated every run. |
| Login budget: failure breaker tripped | 4,785 | Situs | The robot has hit its own sign-in caps most days since Sep 9. |
| Login budget: daily cap / hourly cap | 903 / 837 | Situs | Same story, two more titles. |
| Texts stuck: Clara couldn't answer them | 4,548 | all | Customer-facing, buried in the list above. |
| Rent roll has never been read (6-hour threshold) | 1,462 | Western Slope | Clara refuses rent answers; nobody acted in 31 days. |
| Oct 13 13:26: breaker tripped, Western Slope identity | 1 | Western Slope | The first symptom of the dead key. Same title as the 4,785 above. |
| Oct 14 02:00: "invalid x-api-key" (session refresh, login-lock) | 2 + 15 | both | The first alert text that names the cause, 12.5 hours in. |
534 issues in 60 days, 526 unresolved; zero cron monitors; one rule in effect (new high-priority issues to #alerts), inferred from the posts because our token cannot read the rules.
| Measure (60 days) | Value |
|---|---|
| Alarms that went to ALARM | 390 transitions across 15 alarms |
| Of which CI fleet capacity (on-demand and spot vCPU) | 310 (79%) |
| Subscribers on the alerts topic | one Slack forwarder Lambda; no email, no SMS |
| stage-replicator alarm sitting in ALARM | since Oct 5; 40 Slack posts about it in 60 days, one human reply (Gera, Aug 21) |
| Mac mini dead man's switch | drilled Sep 27; fired Sep 29 and Oct 2, recovered in 12 and 4 minutes; posts to #alerts only |
| Alarm transitions during yesterday's outage | 0 |
Fede's inbox, same 60 days: zero Sentry, alarm or GitHub emails; 37 Vercel "deployment failed" emails, the one alert that reaches him and never means a customer is affected.
Fifty-nine distinct alert paths in the app and its Lambdas, the robot's own alerts, two bots, and 24 jobs on the Mac mini. Nearly all of them end in the same place.
| Path | What trips it | Where it ends | Meant for | Time to a person, measured | Evidence |
|---|---|---|---|---|---|
| AWS CloudWatch alarms (106) | Lambda errors (5 or more an hour), queue backlogs, two flatlines, CI fleet capacity | One topic, one subscriber: a Lambda that posts to #alerts. No email. | "On-call". Nobody is named. | 14 days: 0 of 24 posts touched. 60 days: one reply (Gera, Aug 21). 79% of transitions are CI capacity. | sns-slack-forwarder/handler.ts:60-162; topic subscriptions = 1 |
| Bake alerts (51 kinds: held text, hand-off broken, job failed, workflow health) | Code decides something needs a human | #alerts. 35 "actionable" kinds also email gera@ only, through SendGrid. | "A human needs to see this" | Slack: 2 of 148 customer posts touched in 14 days. Email: one inbox, unmeasured. | bake-alert.ts:516-517, 884-1091; BAKE_ALERT_EMAIL (Vercel and SSM) = gera@ |
| "Page Fede" alerts (Clara did not answer; lead unanswered 15 min) | An empty reply, a crash after an action, a lead row unsettled for 15 minutes | #alerts, no @-mention. The lead one also lands in #agent-trinity. | Fede, by name in the code | Same muted channel. In #agent-trinity Fede answers 14% of posts. | no-reply-alert.ts:115-145; lead-outbox-watchdog.ts:171-191; fede-page-alert.ts:53-77 |
| Sentry (1,830 call sites; 534 issues in 60 days) | Any logged error or warning | Sentry. One rule posts a new high-priority issue to #alerts. Email to the founders on a new issue (inferred; the rules are unreadable to our token). Warnings never leave Sentry. | Nobody: 0 issues assigned | 14 days: 2 of 77 posts touched. The robot's 4,785 breaker warnings never reached Slack. | sentry-filters.ts:158-170; rule 17072804 |
| Smith's triage (reads #alerts) | A post carrying the alert marker | A verdict in the thread, fixes and PRs. "Needs a person" = a blocked page on docs.propflowai.co plus a Decisions row, and a flag call that fails on Smith's Mac ("degraded to in-thread-only"). | Fede, in theory | A verdict on 19% of posts. Yesterday's correct verdict was read by nobody for 19 hours. | alert_remediation_prompt.py:242; trinity_flag.py:64-71 |
| Trinity flag (the one path built to wake Fede) | A session chooses to run the flag script | #agent-trinity with @Fede after a 30-minute grace, only when he is idle 5 minutes, held 10 PM to 7 AM, dropped at 72 hours. No heartbeat on the watcher. | Fede | 14 days: Fede replied to 23 of 165 posts, median 47 minutes (13 minutes over 60 days). Zero flags for the key. Trinity stopped reading #alerts on Oct 12 at 6:34 PM. | trinity_watch.py; flag.sh; plist written 10-12 18:34 |
| The robot's own alerts (login budget, breaker, tenancy) | The robot cannot sign in or hits its own caps | Sentry, at warning level | Nobody | Never reached Slack. 4,785 breaker warnings since Sep 9. | lib/alerts.ts:163-227; lib/loginBudget.ts:581, 932 |
| Robot heartbeat, every 2 minutes | The process is alive | A tile on the admin metrics page | Whoever opens it | Nobody did during the outage. It does not check that sign-in works. | lib/cronHeartbeat.ts; api/admin/cron-heartbeat/route.ts |
| Per-customer "robot cannot sign in" alarm | A job fails on a sign-in error that names a customer | Built to post once in #alerts and edit in place | On-call | OFF everywhere: the switch is absent in Vercel, SSM and the Lambda. | pms-outage-alarm-flag.ts:35-37 |
| Held AppFolio writes (guest cards, tours) | The robot is down | Retried silently for up to 72 hours, then an email to the customer's staff and a Sentry error | The customer's team | Yesterday: 11 writes held up to 20.5 hours; nothing to us. | guest-card-outbox-drain.ts:74-86, 3650-3665 |
| A scheduled job stops running (45 Vercel crons, 24 of them customer-critical; 32 AWS schedules) | Nothing | Nothing: 0 Sentry monitors, the status page shows every cron healthy, Smith's staleness check skips them all. A crash reaches Gera's email for 14 of the 24 critical crons; the other 10 go to Sentry or a run row only. | Nobody | Never | admin-status/sections/crons.ts:66-78; emitters/sentry.ts:152-163 |
| Mac mini jobs (24: Trinity watcher, dead man's switch, call grader, backups) | A job fails or the Mac logs out | The dead man's switch posts to #alerts (fired Sep 29 and Oct 2). The Telegram alarm is broken (wrong keychain name). The call grader writes alerts to a file nobody reads (208 so far). Every job exits 0, even after failing. | Nobody | Never. A reboot or logout stops all 24 until someone signs in. | health-check.sh; launchd inventory (appendix) |
| GitHub workflows (CI, red main, deploy freshness, fleet capacity) | A check fails | #alerts with the alert marker; Smith reverts or fixes | The bots | 456 posts in 14 days, 0 touched by a person. That is the right outcome; the wrong channel. | .github/workflows (appendix) |
| Vercel "deployment failed" email | A build fails | Fede's inbox: 37 in 60 days | Fede | The only alert that reaches him, and never a customer outage. | Vercel project settings |
| Phone, text or push to an engineer | Does not exist. The only calls and texts go to the customer's on-call staff in a resident emergency. | Never | grep for pager, push, Pushover, PagerDuty: 0; emergency-alert-chain.ts |
A failure is silent when nothing wakes a person. With #alerts muted and no pager, that is every class below. The "2 AM" column is what the code does, read path by path, not what anyone hopes it does.
| What fails | What happens today | Anyone at 2 AM? | What the customer sees | Evidence |
|---|---|---|---|---|
| Clara's AI key is rejected (every customer at once) | Each text turn fails into one constant Sentry issue, then the queue's dead-letter alarm posts to #alerts. The "Clara did not answer" page never fires: the first model call had done nothing yet. | No | Leads and residents text and get nothing, everywhere. | process-envelope.ts:1163-1199; inbound-pipeline-health.ts:57-60 |
| The robot cannot sign in (key, caps, breaker) | The alarm built for this is off. Writes retry silently for up to 72 hours, then email the customer's staff. The robot's own alerts are Sentry warnings. | No | Guest cards, showings and work orders never reach AppFolio. Staff hear about it from a "could not create" email days later. | pms-outage-alarm-flag.ts:35-37; guest-card-outbox-drain.ts:74-86; lib/alerts.ts:163-227 |
| AppFolio blocks the sync (403, 429) | A 429 is a log line retried next tick. A lasting one becomes a Sentry warning, "stopped syncing". | No | Clara answers from stale data: a unit already leased is offered. | appfolio-sync/handler.ts:6979-7012; sync-connections.ts:600-641 |
| Twilio rejects or drops texts | Failed and undelivered texts are log lines. Nothing watches delivery rate or a stop in inbound traffic. | No | Clara's texts never arrive, and nobody knows. | twilio/status/route.ts:77-93 |
| SendGrid is down | Bounces post to #alerts; a send error is a Sentry error. The one alert email also rides SendGrid. | No | No email replies; staff notices lost. | sendgrid/event/route.ts:303-326; bake-alert.ts:19-21 |
| The workflow worker dies or sagas stick | Nothing watches the worker (0 alarms on it). Hourly sweeps post symptoms to #alerts and Gera's email. | Gera's inbox, hours later | Renewals, tour reminders and follow-ups stop moving. | temporal-reachability.ts:117-224; bake-alert kinds renewal_stuck_saga, *_workflow_health |
| A Lambda fails, or stops being called | Five errors an hour alarm into #alerts. 22 alarms watch Lambdas that no longer exist. Only two flatline alarms exist; one has been red for nine days. | No | Texts unanswered, data stale or email unread, depending on which Lambda. | 106 alarms, us-east-1 (appendix) |
| A scheduled job stops running | Nothing. 0 monitors, status page always green, the staleness check skips every Vercel and AWS schedule. | No | Outbox drains, notices and sweeps stop. | crons.ts:66-78; detector.ts:104-124 |
| A queue or outbox stops draining | The lead outbox pages #alerts at 15 minutes. Guest cards wait 1 hour (72 on an outage). Others log a Sentry fatal when exhausted. | No | Replies, guest cards and confirmations late or never. | lead-outbox-watchdog.ts:171-191; tour-outbox.ts:315 |
| A mailbox token or subscription expires | A Sentry error per poll round. The status page keeps saying "connected" (Western Slope, Sep 15 to 20: about five days). | No | A prospect's reply sits unread. | company-mailbox-poller.ts:524-527; outlook-subscription-renewer/handler.ts:501-508 |
| The voice vendor degrades | Calls go from Twilio to ElevenLabs directly, so an outage sends us nothing. Nothing watches call volume or silence. | No | Dead air, failed transfers, no post-call notes (Oct 13, 2:55 AM: three calls with no audio). | voice/incoming/route.ts:9-11, 317 |
| A customer's PMS connection dies | A Sentry warning every 30 minutes. The code says a page "needs a Sentry rule: Fede's call"; none exists. If the heartbeat cron dies, nothing notices. | No | Frozen data; nothing syncs. | sync-connections.ts:864-875; sentry-filters.ts:169-170 |
| The alerting itself | Smith's "get a person" step calls a script that is not on its Mac. The Trinity watcher has no heartbeat and swallows its own crashes. The Mac's 24 jobs report success after failing. The one alert email has one recipient. All 104 AWS alarms ride one forwarder program with no email backup; if it breaks, they all go quiet, and its own failure alarm rides the same broken path. | No | Yesterday. | trinity_flag.py:64-71; health-check.sh; BAKE_ALERT_EMAIL |
156 dated incidents from session memory, the docs pages, PR history and Sentry (Aug 15 to Oct 14, plus the long tail that was found inside the window); 131 touched a customer. An alert fired for 13 and a person acted on 8; people, customers and sessions doing other work found 51; 67 have no recorded detection path. "Silent" is the verdict on 80.
| What broke | Unseen for | How it surfaced | Under the proposal |
|---|---|---|---|
| About Apr to Oct 7: the website's demo-request form said "success" and sent nothing | 6 months | Not recorded | P2, needs its own check: a form or feed that goes quiet |
| Apr 29 to Sep 2: Yale 25 Station had no rent roll; Clara told a prospect nothing was available | ~4 months | The go-live review | P1: a customer feed not read for 30 minutes |
| Aug to Sep 1: Camellia rent not billed for 14 residents (skipped, expired and move-in leases), about $21,000 by estimate | Weeks | A session doing other work | Not an alerting problem: a billing check against the rent roll |
| Aug 20 to Sep 3: Camellia escalation reminders went to a PropFlow hello@ address, not the team | 13 days | Fede, by hand | P2, needs its own check: a notice sent to an address nobody reads |
| Aug 21 to Sep 4: renewal escalations had no owner, clock or re-page | 14 days | A session | P2, needs its own check: an escalation with no owner after a day |
| Aug 22 to Sep 1: Camellia transfers rang into a line nobody answered | ~10 days in the window | A customer told us | P2, needs its own check: a transfer with no pickup |
| Jul to Sep 23: renewals stuck in "prepare failed" with no deadline; a resident was texted an offer that did not exist | 64 to 72 days | Not recorded | P2, needs its own check: a stuck row older than a day |
| Jul 28 to Aug 20: a stage copy of production sent real email from Camellia's mailbox | ~23 days | A session chasing bounce floods | Not an alerting problem: an environment guard |
| Aug 4 to 18: an escalated thread never released; a tenant two days from move-in got a canned reply | 14 days | The tenant's unanswered question | P2, needs its own check: escalated thread with no human reply |
| Sep 1 to 7: cross-company data visible to an admin with zero properties | 6 days | Fede, from a screenshot | Not an alerting problem: a code floor |
| Aug 31 to Sep 6: the Trinity seat ran stale code; every reply failed (26,000 failures) | 6.4 days | A session | Dead man's switch on the bots |
| Sep 15 to 20: Western Slope's leasing mailbox stopped being read while the status said "connected" | ~5 days | A fix for something else | P1: mailbox not read for 30 minutes |
| Sep 18 to 22: Western Slope forwarded its main line into Clara; transfers looped and callers dropped | ~4 days | A session | Not an alerting problem: prove routing with a real call |
| Sep 21 to 25: callers naming a community got no guest card | ~4 days | Fede sent the call ids | P2, needs its own check: a call that names a home ends with no card |
| Aug 22 to 25: a self-mail gate dropped every Camellia website-form lead | ~3 days | Not recorded | P2: the lead-unanswered watchdog built since |
| Oct 1: a Camellia transfer rang out, Clara said "office closed", no note saved | ~8 days | A session reviewing calls | P2, needs its own check: no pickup and no note |
| Oct 7 to 8: the Western Slope robot broke on an AppFolio self-redirect on the first self-guided tour night | ~1 day | The retro | P1: no successful robot cycle for 30 minutes |
| Oct 12 to 13: JP&Co guest cards unreadable (403) after a robot-user switch | ~7.4 h to notice, 23 h to fix | A session | P1: same rule |
| Sep 29: Situs rate-limited within minutes of connecting; sync paused | ~12 h | Not recorded | P1: same rule |
| Sep 26: the Redis store behind the login budget hit its request cap; robot logins blocked at Situs and Western Slope | ~11.8 h | Sentry only (14,535 errors) | P1: same rule |
| Sep 20: a headless session switched Western Slope's Clara off; a lead got no reply | 1.6 h or more | Fede: "didn't we turn it on?" | P2, needs its own check: a customer switch changed by anyone but Fede |
| Oct 13: the robot's AI key rejected; 11 writes held | 37 min to post, 20 h to fix | #alerts, no page | P1: same rule, paged at 1:56 PM |
| Oct 13, 2:55 AM: the voice vendor degraded; three calls had no audio | Never alerted (40 min) | A session, next day | P2, needs its own check: a call with no audio |
| Source | What was noisy | Count |
|---|---|---|
| Sentry, all issues | Nobody owns them; 19 issues carry 91% of events | 534 issues, 0 assigned, 526 unresolved |
| Sentry, one robot condition | "Account budget exhausted" re-fired for 34 days | 25,311 events in one issue |
| Sentry, one config gap logged as fatal | "PM reminder has no recipient", Sep 12 to 16 | 12,436 fatal events |
| Sentry, pre-launch and test data | Situs "phone coverage 0%" before go-live; test-named issues | ~8,800 events |
| Sentry, quota | Events dropped on the worst outage days, so the log went blind when it mattered | 6,489 dropped |
| Smith in #alerts | Its own "PR waited N min" self-checks (Sep 11 to 12); 22 overnight mentions | 97 of 849 posts |
| Per-job robot alarms | One sign-in cause, every job alarming every run (Sep 9 to 11) | 10 pings, 1 cause |
| Bad-merge flood | One bad merge, Sep 23 | ~19 alerts in 7 min |
| Trinity flags before the rewrite | Flags raised and never closed | 23 of 25 in 30 days |
| #spending | An 8-hourly CI status script; the same six AWS anomaly ids nightly; eight leaked test pages | 87 posts in 2 weeks |
| Test runs pinging Fede | A scripted tenant at Willows raised review holds that each pinged him (Oct 6; fixed) | ~11 in one morning |
| CI merge guard | Trains ejected, green PRs disarmed, spot machines interrupted | 281 trains in 3 days; 12 green PRs disarmed |
| Clara notices to customer staff | 14 emails in 3 seconds (Aug 31); 6 stale reminders (Sep 7); 11 escalation emails and 10 texts in 5 hours, 4 needed (Oct 7 to 8) | 4 episodes |
| Alert | When | What it caught | Then what |
|---|---|---|---|
| Bounce alert to #alerts | Proven Aug 16 | A real bounce, naming the recipient and reason | Built after the Aug 15 incident |
| Login-budget breaker, triaged by Smith | Sep 9, 4:19 PM | Robot lockout within minutes | Misdiagnosed as AppFolio blocking us; 10 pings for one cause |
| Sentry plus Smith, missing PM recipient | Sep 12 | Western Slope had no PM email (pre-launch) | Within the hour; 10 issues for one gap |
| Lead-event alarms | Sep 23, 12:16 PM | A bad merge flooding lead events | Reverted in minutes; nothing sent |
| Calendar-binding alarm | Oct 6, 6:55 PM | Western Slope's calendar binding gone | Fired; the response (removing a tour host) was wrong |
| Smith failed-write posts | Oct 7, Oct 14 | 6 and 5 failed AppFolio writes | Surfaced; program open |
| Robot breaker to #alerts | Oct 13, 2:03 PM | The dead key, 37 minutes in | Triaged at 45 minutes; no page; fixed 20 hours later |
| Voice drift check | Aug 24, 6:15 AM | The voice tool sync read only the first page; Camellia's voice agent was missing tools | Fired; the finding sat about 4 days |
| Renewal-call drift check | Aug 28, 3:25 AM | Camellia's renewal-call script a week behind | Fired; the fix waited on a person |
| Sentry and the nightly canary | Sep 10 to 11 | Robot logins broken by our own change | Fired three nights, 5 copies per run; fixed after 16 hours |
| Camellia byte-identity replay | Sep 28, about 3 PM | A shared production prompt edited by hand | Fired; fixed (time not recorded) |
| Leasing workflow health sweep | About Oct 8 | Four Western Slope outreach workflows terminated | Fired and re-paged; repaired |
| CI before merge | Measured Oct 5 | About 14 real breaks a day | Caught before production |
| Review hold | Sep 22 | A generic first email to a Western Slope prospect | Caught before send |
| When (MT) | What broke | Who felt it | Found by | Fixed after | Alerting verdict | Source |
|---|---|---|---|---|---|---|
| Jul 28 to Aug 20 | Stage copy of prod sent real email from Camellia's mailbox | Camellia's mailbox emailed fake addresses; "Undeliverable" bounce floods Jul 28, Aug 14, Aug 19-20 | a session stumbled on it (Aug 20) | n/s | late (prod "capture drift" alerts fired on it but were read as a prod defect) | session memory |
| ~Aug 1 to Aug 15 09:37 | Seven lease-countersign pollers each logged in to AppFolio hourly | internal only (~740 sign-in codes in two weeks, 150 on Aug 14; 16 in one hour) | a human noticed (Fede saw 16 codes in an hour) | rolled out 09:37 Aug 15 | silent (inferred) | page |
| Aug to Sep 1 | Camellia rent not billed (skipped leases, expired leases, move-ins) | ~$21,000 (est.) unbilled across 14 residents: 5 skipped; 7 of 8 expired leases (~$8,650 est.); six move-ins (~$11,800 est.) | a session stumbled on it (Sep 1) | n/s | silent | page |
| Aug 4 to Aug 18 (canned reply Aug 17 22:29) | Escalated-thread gate swallowed a tenant's key-pickup question with a canned reply; escalations never released | a Camellia tenant 2 days from move-in got no answer for hours; a second tenant stonewalled from Aug 17; two escalations sat 14 days | a human noticed (Fede spotted the thread, Aug 18 daytime) | team key-pickup email a PR same day; code fixes still pending at RCA time | silent (no email, Slack or re-notify for an escalated thread) | page |
| Aug 13 to Oct 14 (grouped) | Our own watchers and bots silently dead (20+ cases) | internal only (no customer message traced to these, inferred) | a session stumbled on it (mostly) | per case; the runtime-update gap is still open Oct 14 | silent (the one warning that did fire, the updater's, cried wolf: it warns every session and is ignored) | Aug 13-16 review-bot token dead; Willows harness phases dead |
| Aug 14 09:01 to Aug 15 14:46 | Booking-honesty guard replaced real replies with "Nothing is scheduled yet" | 3 real Camellia prospects got the canned line instead of their answer (Aug 14 09:01; Aug 15 10:59 and 11:19) | a human noticed (Fede: "misbehaving terribly") | ~29 h 45 m (14:46); revert a PR at 13:24 (a PR live 14.4 h), revert a PR at 14:31 | silent ("40+ bench events buried the 3 real ones") | page |
| Aug 14 21:40 to Aug 15 ~10:10 (CONFLICT: the booking-guard page says Aug 15 2:10-10:10 AM) | Prod AI key hit its per-minute rate limit; replies fell back to the canned "our team has it" | Camellia prospects who wrote overnight got only the canned fallback when requests burst ("bursts fail, single requests pass"; count n/s) | a session stumbled on it (overnight session; also posted to #alerts) | ~12 h 30 m (est.; still failing 09:21, ended ~10:10); consumer never identified | late (inferred: posted to #alerts, nobody could act overnight) | page |
| Aug 15 04:30 | An early move-in resident was treated as a prospect | a resident's (Camellia, inferred) 04:30 towing call got a 35 s transfer, then voicemail | n/s | ~1 day | silent ("No alert fires") | docs /a/<name>-resident-visibility-2026-08-15 (name in |
| Aug 15 11:57 to night | Microsoft blocklisted SendGrid's shared IPs; Camellia team notices bounced | 4 Camellia team-notice bounces (11:57; two at 13:00, one of them a "waiting on your signature" email never redelivered; 15:53) | a session stumbled on it | same night (delist request and SendGrid ticket; bounce alarm proven Aug 16) | silent ("bounce handling only watches renewal emails") | page |
| Aug 15 | Manager-reminder gate read the tour window as office hours | Camellia's manager got a Saturday email while the office was closed; 27 of 34 manager emails went out off-hours | n/s | n/s | n/a | page |
| Aug 15 22:42; Sep 12; Sep 28 to Oct 1 (grouped) | Fleet machine overload, SSH drop and disk exhaustion | internal only: Aug 15 SSH drop took 4 of 7 workspaces; Sep 12 ten sessions with no admission control filled swap and froze the Mac mini that runs Trinity, and the Claude account pool ran dry; Oct 1 free disk fell from 189 GB to 19 GB | a human noticed (Fede, Sep 28 disk; other cases inferred) | load guard, hung-run reaper, disk janitor; Sep 28 cleanup took free space from 17 GB to 154 GB in 3 h 40 m | late (the Oct 1 disk alert fired at 38 GB free, "too late"; three cleanup steps had been "quietly doing nothing for days") | page |
| ~Aug 16 | A relayed staff answer was held by the 7-in-7 send cap | n/s (not measured; Camellia, inferred) | a session stumbled on it (Willows replay) | a PR | silent | page |
| Aug 16 to Oct 8 (grouped) | Merge and fleet process misfires (13 listed) | mostly internal; one customer exposure: a Camellia vendor-call greeting change live 59 min with no switch (Sep 7, a PR, revert a PR) | a human noticed (Fede, mostly; inferred) | reverts or closes | silent | session memory |
| Aug 19 | A reply falsely said "application link sent" (shipped green, failed between units) | one Camellia prospect told a link was sent when it was not | n/s | n/s | silent | page |
| Aug 20 to Sep 3 | Camellia escalation reminders went to a PropFlow hello@ address, not to Camellia staff | a Camellia prospect who wrote Aug 20, 23 and Sep 3 got no reply until Sep 3 (13 days); three others waited 14, 9 and 17 days | a human noticed (Fede, by hand, Sep 3) | n/s (renewals page counts 9 fixes for escalations routed to the wrong inbox) | silent | page |
| Aug 20 08:30 to 16:08 (page in CDT, converted) | An ungated staff toolbox auto-replied to Camellia staff | JP&Co staff got eager Clara replies (08:30; 09:33-09:36; recurred 16:08); a 10:19 filler non-answer to an applicant | a customer told us (JP&Co staff); Fede caught the recurrence | 1 h 44 m to the first fix (flag off, structural fix merged 10:14); recurrence fix n/s | silent | page |
| Aug 20 | Bounced reply classed "unknown"; Zillow lead went dark | a lead (Camellia, inferred) got no reply | a human noticed (Fede) | ~1 day (a PR merged Aug 21) | silent | session memory |
| Aug 21 to Sep 4 | Renewal escalations had no owner, clock or re-page | a Camellia resident's (inferred) paper-lease request got 14 days of silence; her lease ended Aug 31 | a session stumbled on it (Sep 4) | n/s | silent | page |
| Aug 22 to Aug 25 | Self-mail gate (a PR) dropped website-form leads | Camellia website-form leads got no reply (first real casualty Aug 25 00:42; count n/s) | a session stumbled on it | a PR merged Aug 25 ~13:30; end-to-end proof Aug 26 | silent (source: "silently dropped") | page |
| Aug 22 to Sep 1 (problem since May 20) | Camellia transfers rang into an unanswered line | 19 transfers, 7 failed, 5 callers dropped with nobody told (failures Aug 22, 27, 29, 30 including the alarm company, three on Sep 1) | a customer told us | fixed on Willows only | silent ("black hole, then self-silenced") | page |
| Aug 22 06:16 to Aug 28 12:40 | Voice tool sync read only the first page; Camellia voice agents grew from 64 to 155 tools | Camellia voice agents ran with 155 tools instead of 64 (effect on calls n/s) | alert (voice drift check, Aug 24 06:15) | code fix 3 d 3 h after the alert; cleanup Aug 28 12:40 (4 d 6 h after) | helped (but the finding sat ~4 days) | page |
| Aug 22 | Voice told a caller "you're all set, tour rescheduled" when the slot was taken and nothing was booked | a real caller told a reschedule was booked (customer not named) | n/s | evidence PR a PR (16:58) | n/s | a PR (PR history) |
| Aug 22 20:25 | Voice had no Saturday availability lookup; a stage-cue leak caused dead air (2.9 s, 2.0 s) | none ("zero real customers") | n/s | rollback within 8 min 15 s | helped (later, via the listen gate) | page |
| Aug 24 to Oct 12 (grouped) | CI red main: 163 incidents Sep 18 to Oct 2 alone (count) | internal only (merges frozen, armed PRs disarmed; 6 product reverts were caused by red main, none reached customers, inferred) | alert (CI) | median 0.5 h, p90 4.2 h (Sep 18 to Oct 2); full-suite-only-on-main left main red 61% of the time vs 13% (revert a PR); 4 of 73 red-main fixes were flakes | helped, but mislabeled and loud: since Oct 1 20:03 UTC the queue's fail-fast cancels its own run (Oct 2: 0 failed vs ~1,377 cancelled; 94% of 180 sampled cancellations held a real failure); ~68 "main is red" posts a day | page |
| Aug 24 21:02 | Clara email leaked internal reasoning and other people's details | a Camellia prospect saw internal notes and two other customers' names; a resident's guest-card note was pasted into another prospect's email (repo ADR, same incident, inferred) | n/s | n/s | silent ("an absent marker raises no alarm") | page |
| Aug 25 to 26 | Blocked calendar slots bookable; empty guest cards; nudges on staff-owned threads | Camellia: tours offered in blocked slots, empty guest cards, a prospect got a duplicate tour | n/s | fixes a PR, a PR, a PR, a PR live-verified by Aug 26 00:10 | n/s | session memory |
| Aug 26 14:24 to 16:17 | A kill switch (a PR) turned off ALL PM paging at every real property after 2 false Camellia pages | for ~1.9 h no property manager at any real property could be paged | n/s | ~1.9 h (revert a PR at 16:17) | silent (paging was off with no alarm, inferred) | a PR, a PR (PR history) |
| Aug 28 03:23 | Frozen Camellia renewal-call script was a week behind | Camellia renewal calls ran on a week-old script (calls affected n/s) | alert (drift check, 03:25) | n/s | helped | page |
| Aug 28 12:33 to Aug 29 | Escalated thread kept talking: email and timer sends skip the escalation gate (plus a deploy race, a PR) | a Camellia prospect got a walk-back of a staff-approved voucher answer, invented claims at 12:35, an "apply now" timer email at 12:45, and a settled answer overwritten at 02:31 ("I need humans to come back"); 23 real escalated Camellia conversations exposed to the same gap | a customer told us (inferred) | correct answer reached the prospect 20:26 (7 h 53 m after the complaint); 9 fixes merged overnight; the mute was dormant | silent ("zero of the sixteen mechanisms" would have alerted) | page |
| ~Aug 28 | Clara replied on an internal staff forward | JP&Co regional PM (Kenya) got Clara replies on an internal thread; says she "sometimes gets emails from Clara" | a customer told us | n/s | silent | session memory |
| Aug 29 | Person-mute had 5 verified leaks | a Camellia resident under a full-mute order still got replies | a session stumbled on it (adversarial verification) | n/s | silent | session memory |
| Aug 29 to Oct 7 (grouped) | CI and deploy infrastructure stalls and silent limits | internal only (prod ran stale code: a Lambda ran old code 30+ min after a flaky push CI, a PR; the renewal worker ran ~2 h behind web, Sep 30) | a session stumbled on it (mostly; the deploy-freshness alert caught the Sep 22 and Sep 25 build collisions) | per case: Turbopack collisions blocked promotes 39+ min (a PR, a PR); runner launch failures for hours (a PR); GitHub 403s ~2 h; queue emergency OFF Oct 1 20:27 | silent (the deploy alarm needs 120 min, the worker lag was ~2 h; the merge gate reads only the first 100 checks; the watchdog drops the oldest PRs near its cap of 100) | page |
| Aug 30 | A merged-away duplicate lead came back (second time) | Camellia staff saw a duplicate lead again (inferred) | n/s | n/s | silent | page |
| Aug 30 09:49 to 10:06 | Wrong escalation with a made-up unit offer; a full reply discarded | Fede's own test conversation on Camellia's live line: the property manager got a wrong escalation, the "prospect" got silence (same defects applied to real prospects, inferred) | a human noticed (Fede's test) | none fixed as of Aug 30 evening | silent | session memory |
| Aug 30 16:39 | Fair-housing gate rewrote a correct reply | a Camellia hot lead was told to wait two weeks | a human noticed (an operator) | a PR and a PR (gate rewrites retired) | silent (the judge flagged it in 15 s and told no one) | page |
| Aug 31 | "Ask" lane emails sank to the wrong inbox | internal only (5 broken emails to Fede at Willows; no JP&Co staff exposed) | a human noticed (Fede, inferred) | n/s | silent | session memory |
| Aug 31 09:01 | Staff email storm | internal only (test staff inboxes before turn-on, per agent b; the memory file calls them staff): 14 emails in 3 s, one email 6 times, an emergency follow-up 40 s after the emergency page; 106 staff emails Aug 25 to Sep 1 | a session stumbled on it (Sep 1 inspection) | n/s | silent (SendGrid content retention OFF, so bodies cannot be reviewed) | session memory |
| Aug 31 11:50 to Sep 6 20:36 (time zone not stated) | Trinity seat ran stale code | internal only (every Trinity reply failed: 26k failures, 44 stuck workflows) | a session stumbled on it (inferred) | Sep 6 (restart) | silent ("nothing alerted") | session memory |
| ~Aug 31 to Sep 4 | A formal notice to vacate ("September 30th") fired nothing: no notice record, no inspection, no AppFolio draft | a holdover resident's (Camellia, inferred) notice to vacate was not acted on | a session stumbled on it (Sep 4) | a PR did not close it | silent | page |
| ~Aug 31 (n/s) | The lapsed-lease renewal branch skipped eligibility | residents behind on rent got a month-to-month quote (Camellia, inferred; count n/s) | a session stumbled on it (renewals audit, inferred) | a PR (dark) | silent | page |
| Aug 31 19:51 | Promise checker used the UTC day; false "tour not on file" escalation | customer staff (Camellia, inferred) got a false escalation; 12 or more evening-time defects were live | n/s | n/s | cried wolf (plus silent defects) | page |
| Sep 1 to Sep 10 (CONFLICT: Sep 9 vs Sep 10 for the session-lookup failure) | Cross-company isolation gap: a failed session lookup was treated as "platform staff, no restriction" | Camellia data visible to an admin with zero properties (Sep 7); a zero-property company saw platform-wide rent ($299,100) and overdue ($37,047); Ask Clara returned other customers' data | a human noticed (Fede's WS walk-through; screenshot) | a PR and a PR live ~16:05 Sep 10; a PR and a PR held (Fede: "stop the PRs"), now Gera's | silent | page |
| Sep 2 21:42 to ~22:12 | A second renewal starter (120-day window) was switched on | 7 Camellia residents got renewal offers early (5 of them four months early); 10 pages sent twice; 3 vague emails | n/s | switched off in ~30 min; deleted by a PR (Sep 4) | cried wolf | page |
| Sep 3 | A June 26 call was re-graded as new and paged as a broken promise | customer staff (Camellia, inferred) got a false broken-promise page | n/s | a PR forensics only; a PR dark | cried wolf | page |
| Sep 3 to 4 | Voice dead air (30.6 s) | 2 of 1,248 calls had long dead air | a human noticed (Fede's Yale test call) | n/s (suspected ElevenLabs-side change) | silent (Sentry holds zero ElevenLabs issues) | session memory |
| as of Sep 3 | 66 of 111 renewal records lacked their AppFolio reference | internal only (data gap; effect n/s) | a session stumbled on it (renewals audit) | n/s | silent | page |
| as of Sep 4 | Renewal offers with no signer email were refused, then parked behind a 65-day marker | 5 residents (Camellia, inferred) never contacted about renewal | a session stumbled on it (renewals audit) | n/s | silent | page |
| ~Sep 4 to Sep 5 08:46 | Application sync skipped 2 Camellia applications while reporting green | 2 Camellia applications missing from Clara until Sep 5 08:46 | n/s | Sep 5 08:46 | silent (sync reported green) | agent b only |
| Sep 5 23:13 to Sep 6 08:14 | WS prototype voice line dead on connect (~1 s, ElevenLabs error 1008) after self-booking tools were bound | WS prototype line callers got nothing for ~9 h (pre-launch; calls in the window n/s; 9 real calls had worked earlier) | n/s | ~9 h (revert a PR at 08:14) | silent (inferred) | repo docs/runbooks/elevenlabs-voice-agents.md#standalone-pro |
| Sep 7 22:00 to Sep 8 12:03 | Reminder migration (a PR) re-sent reminders | 6 stale "New application to review" emails to Camellia staff | n/s (Fede ordered a stop) | ~14 h (restored 12:03 Sep 8; fix a PR) | silent (inferred) | session memory |
| Sep 9 to Sep 16 | Caller identity flickered off every 5 minutes (from a PR) | one resident's 3 calls lost recognition; 3 of 6 missed-call emails damaged (agent b) | a session stumbled on it (Sep 16) | n/s | silent | page |
| Sep 9 15:06 to ~17:00 | Robot login redirect loop on its saved browser profile (Outage A): 20 attempts in 37 s, 21 throwaway profiles | internal only (JP&Co and Situs robot logins locked; "nothing is lost"); 23 sign-in codes to Fede's phone in 53 min, 28 to the WS account's number the night before | alert (login safety limit, 15:06) | ~1 h 54 m | helped | page, Sentry 16J |
| Sep 9 21:52 to Sep 13 | AppFolio 429 storms: two sync workers on one account, then no per-account request budget on connect | AppFolio rate-limited our sync at WS (connect Sep 11) and JP&Co (Sep 12); work orders parked to stop the storm (~258 requests a minute vs AppFolio's ~28); Sep 13 WS re-import hit 429s on ~48 of 69 properties, a 75-unit gap (agent b) | n/s | protections hand-pressed per account | silent (Sentry logged 25,311 events from Sep 10 23:42 with no page; the coverage checker sees only dispatched connections) | page, Sentry 17B |
| Sep 9 22:13 to at least Sep 10 09:00 (verified fixed Sep 11) | WS inbound-text webhook empty (Twilio 11200); texts to the line went nowhere | texts sent to the WS line went nowhere (pre-launch; senders n/s) | n/s ("nothing we watch noticed") | verified Sep 11 | silent | WS tracker row spl-inbound-texts-dead |
| Sep 10 15:58 to Sep 12 ~03:00 | Robot logins broken by our own change (Outage B), misdiagnosed as "AppFolio blocking us" | internal only (no message lost; robot work stalled at JP&Co, WS and Situs; Camellia caches went stale) | alert (Sentry within minutes; canary red three nights), but "No person was actually on the hook" | JP&Co ~16 h 46 m; WS ~23 h 44 m; Situs ~35 h (~03:00 Sep 12) | cried wolf (5 copies per run, 10 pings for 1 cause, 5 jobs each alarmed every run, the cause never paged; 03:30-07:46 alerts got automated summaries only; Sentry dropped 5,164 events over quota Sep 11-14) | page |
| ~Sep 12 to at least Sep 30 | Work-order, renewal and vendor sync jobs disabled to stop the 429 storm | "stale lease end dates at every customer" for ~18 days (est.) | n/s | n/s (still disabled at Sep 30) | silent (inferred) | page |
| Sep 12 01:00 to Sep 16 | WS had no PM email; reminders resolved to nobody | WS PM reminders and application-review notices reached nobody (WS pre-launch) | alert (Sentry, Smith) | n/s | helped but noisy: 10 Sentry issues for 1 gap, 12,436 fatal events | Sentry 18B |
| Sep 12 20:51 to Sep 17 | AWS Config recorded every CI machine after self-hosted runners went live (plus a sync job lock written and deleted ~700 times a minute since Sep 10, a PR) | internal only (AWS Config $1,266.69, 65% of September usage; Sep 15 whole-account usage $557, Config $378; offset by credits, cash impact $0) | a human noticed (Fede pasted the AWS email, Sep 16 22:36) | Sep 17 (recorder cut to 12 resource types) | late ("Nothing automated carried the news"; the spend dashboard said "$0" while $557 a day burned; 16 cost anomalies detected, the $100 floor admitted 2; the budget alarm feeds an SNS topic with 0 subscribers) | page |
| Sep 13 | Scattered-site units went missing after sync (a house address in the unit label read as a stray header) | n/s (units missing from Clara's view; company n/s) | alert (Sentry unit-count drift, inferred) | n/s | helped (inferred) | repo docs/runbooks/sentry-appfolio-unit-count-drift.md#what- |
| Sep 14 | Inbound SMS queue backlog (visibility timeout 1.2x the consumer's) | inbound texts delayed, oldest message 47.5 min; one message dead-lettered (customer n/s) | a session stumbled on it (AWS cost audit, Sep 17) | n/s (fix proposed: raise the visibility timeout) | silent | page |
| Sep 15 | WS mailbox consent and bind failed | WS leasing mailbox could not connect; Kat's own consent hit a dead end | n/s | n/s | silent (inferred; back office showed "no record of who", logs expire in hours) | page |
| Sep 15 to Sep 20 15:47 | WS leasing mailbox push subscription expired while status said "connected" (renewer a PR dropped org mailboxes) | WS prospect replies were not being read until the fix | n/s | fix a PR verified Sep 20 15:47 | silent (status stayed "connected"; Sentry held 116 "subscription create failed 401" errors Sep 10 to Oct 13, no page; the renewer failed 12.9% of runs, p95 59 s of a 60 s timeout) | page, Sentry 16Y |
| ~Sep 16 | WS knowledge gold set only half loaded (7 exceptions missing) | a WS caller got wrong tour info (agent b) | n/s | n/s | n/s | WS tracker row ws-knowledge-verified-live-2026-09-16 |
| Sep 16 | PM-reminder tick had no per-tick cap | ~100 emails into WS's leasing inbox in 45 s | n/s | a PR | n/s | WS tracker row ws-pm-reminder-per-tick-cap-2026-09-17; a PR |
| Sep 16 14:58 to 16:44 (change deployed 13:39) | All-customer voice outage: an empty post_dial_digits value ended every call ~3 s after the caller spoke | 19 failed calls, 0 got through; 11 were from 2 Camellia callers (the rest our own test calls) | a human noticed (routine WS test calls) | 1 h 46 m (revert a PR merged 16:41, synced 16:44; a PR; guard a PR) | silent (no monitor for ElevenLabs' 'All LLMs have failed' error or for a run of short calls) | page |
| Sep 16 21:32 to Sep 17 13:15 | A stale hand-written note rode into WS staff tour-confirmation emails | WS staff tour emails carried a stale note; emails then held Sep 17 11:35-13:15 | a session stumbled on it (deep inspection of a WS property) | a PR, a PR; the hold was patched 5 times in 8 days (a PR, a PR) | n/s | page, Sentry 198 |
| Sep 17 15:32 | Front-desk transfer loop (precursor of C074) | n/s (WS callers, inferred) | n/s | a PR | n/s | a PR (PR history) |
| Sep 18 to Sep 24 | Callers naming a home by community or street got no guest card | 7 WS leads in 5 days got no guest card; 5 real misses where the caller named a community (Sep 18 08:13 and 13:05; Sep 22 12:04; Sep 24 17:28 and 17:48); a Sep 21 13:48 caller named a home and still got none | a customer told us (WS email, Sep 21 16:01) | named-home case proven Sep 22 night; the no-home case is not fixed, by decision (Fede, Sep 21) | silent ("never enqueued, silent"; an alert added Sep 22, a PR, was removed Sep 23, a PR, as noise; a PR's test asserts "no card") | page |
| Sep 18 09:28 to Sep 25 13:08 | Transfers to an empty number (Twilio 21201) dropped real callers | a WS caller dropped (Fede: "this is bad"); a contractor refused; recurred Sep 24 with a client caller | a human noticed (Fede, inferred) | ~7 days (a PR Sep 25 13:08; a PR was a log line only) | silent | page |
| ~Sep 18 | An unsettled call forward left no record | a 5-minute call was lost with no record (WS, inferred) | n/s | a PR | n/s | a PR (PR history) |
| ~Sep 18 | WS AppFolio notes returned success but never landed | WS notes missing in AppFolio while we recorded success (count n/s) | n/s | robot #392 | silent (false success) | robot #392 |
| Sep 18 (agent b: Sep 19) | 31 of 145 WS people records duplicated | duplicate WS people records (who saw them: n/s) | n/s (agent b: reached Sean) | a PR, a PR | n/s | page |
| Sep 18 13:36 | A tenant waited 37 s in the call center's language menu and hung up | a WS tenant (inferred) hung up after 37 s in the menu | n/s | a PR, a PR | n/s | a PR, a PR (PR history) |
| Sep 18 ~17:58 to Sep 22 | WS forwarded its main line to Clara; transfers to it looped | WS callers dropped on transfer (3 calls on Sep 22) | a session stumbled on it (Sep 22) | rule changed Sep 22; routing re-proven with real calls (date n/s) | silent (inferred) | session memory |
| Sep 18 17:58; Sep 27 06:14 | Test calls reached WS's real call center; fake guest cards written into WS's AppFolio | WS's call center got a fake voicemail (via a PR); fake guest cards in WS's AppFolio on Sep 18 and Sep 27 06:14 that cannot be deleted | n/s | overnight test traffic removed (date n/s) | n/s | page |
| Sep 19 to 21 | Clara offered occupied units as available | Camellia prospects told occupied units were open; staff (Erika) corrected by email 2 min later | a customer told us | a PR, a PR, a PR | silent (inferred) | session memory |
| Sep 19 16:29 to Sep 21 ~12:00 | One WS prospect had three days of misses: a rented home offered, $27 vs $40 fee | a WS prospect was offered a rented home and quoted the wrong fee | n/s | contained in ~43 h 30 m | silent | page |
| Sep 19 night to Sep 21 10:49; Sep 21 09:00 to Sep 23 20:00 | Clara skipped every real WS lead: the reply path read the card creator, which was never saved | the 2 real WS leads that weekend got no first email; a Sep 21 09:00 Zillow lead was found Sep 23 20:00 (~2 d 11 h); 30-day count: 5 cards never assigned (WS 3, JP&Co 2) | a session stumbled on it (production logs) | a PR merged 10:30, deployed 10:49 Sep 21 | silent (no alert for a skipped real card until the "FIRST EMAIL ERRORED guest_card_created_by_unknown_exhausted" page) | page |
| Sep 20 to Oct 9 16:30 | Multi-home renters got several guest cards | WS: 7 extra cards for 5 renters, 6 extra schedules, 10 split threads | n/s | cleanup Oct 9 16:30 | n/s | page |
| Sep 20 08:56 | A headless session switched WS Clara-live off | a 10:31 WS lead got no reply and no review notice | a human noticed (Fede: "didn't we turn it on?") | n/s | silent | session memory |
| Sep 20 16:24 | Draft compose crashed; drafts died quietly | 2 WS leads got no first email | n/s | n/s | silent (recorded only as "send-failed" in the database; held-draft alarm a PR still owed; the first-email engine's failure alert went to an unwatched inbox until a PR, Oct 1) | session memory |
| Sep 21 | Real call to the WS line refused | callers like Fede refused (inferred) | a human noticed (Fede's call) | a PR at 12:10 | silent (inferred) | session memory |
| Sep 21 | Front-door property resolution read a short reply as cold and opened a duplicate conversation | a WS prospect's reply split into a duplicate conversation (inferred) | n/s | harness a PR (13:12) | n/s | a PR (PR history: "two production incidents of 09-21&qu |
| Sep 21 onward (n/s) | ~21 occupied WS homes listed as vacant | a real caller heard an occupied address; 8 sends went out before the never-offer list | n/s | never-offer list (date n/s) | n/s | page |
| Sep 21 | Staff handoff dropped (RingCentral answered in 1-2 s; the press-1 window expired) | WS callers handed to staff were dropped (count n/s) | a customer told us (WS issue-list email, 16:01) | a PR live 16:14 (13 min after the email) | silent (inferred) | page |
| Sep 21; Sep 23 11:17 | False and empty voicemail notices | WS staff got a false "voicemail" notice for a connected maintenance forward (Sep 21) and empty voicemail notices (Sep 23 11:17, agent b) | n/s | a PR (Sep 23) | silent (inferred) | session memory |
| Sep 21 to 22 | Two ring-list regressions in three days | a WS call went to Kat's voicemail and the next person never rang | n/s (reached WS staff) | n/s | silent (inferred) | session memory |
| Sep 22 | A known WS tenant got a guest card | a tenant filed as a prospect (inferred) | n/s | n/s | silent (inferred) | session memory |
| Sep 22 | Hang-up notices with partial context unclear | WS staff complained about unclear follow-up messages | a customer told us | n/s | silent (inferred) | session memory |
| Sep 22 | Alarm-panel robocall became a "call them back" email | Camellia staff got a request to call back an automated alarm | a human noticed (Fede: "just take the message") | n/s | silent (inferred) | session memory |
| Sep 22; Sep 23 | Bad first-email drafts: a generic draft (Sep 22); a draft greeting the renter twice entered the hold unflagged (Sep 23) | internal only (held for review, not sent) | caught by the review hold (Fede) | root-cause fix + regression test (Sep 22, PR n/s); a PR (Sep 23 22:50) | n/a (review gate caught it) | session memory |
| Sep 23 12:14 to 13:24 | Lead-event flood from a PR (an event for every historical inquiry) | none (nothing sent; the 17 parked WS leads were year-old backfill inquiries: 16 "inquiry not found", 1 identity clash) | alert | revert a PR merged 12:48, live 13:24 (~1 h 10 m) | helped (noisy: ~19 alerts in 7 min for 1 cause) | page |
| Sep 23 12:16 to Sep 28 | A WS applicant's first email never sent: the lead parked after 5 claim-race attempts in 32 min | a real WS applicant got no first email | a session stumbled on it (Sep 28) | a PR (Sep 28) | silent (silently parked) | repo docs/planning/incidents/2026-09-28-lead-outbox-claim-ra |
| Sep 23 ~18:00 | "Property not identified yet" lead email with a raw dump | WS team got an unstructured lead dump with a manual follow-up ask | a human noticed (Fede: "This is bad behavior... we shouldn't be noisy") | n/s | silent (inferred) | session memory |
| Sep 24 08:04 to 10:37 | Guest-card sync failed every card at one property (appfolio-409) | that property's cards unsynced ~2.5 h (company n/s) | alert (Sentry only, 153 errors) | cleared 10:37 (cause n/s) | n/s (no doc found) | Sentry 1B3 |
| ~Sep 24 | A WS applicant left unanswered (escalation had no owner) | a WS applicant got no answer (agent b) | n/s | a PR (escalation-owner fix) | n/s | agent b only; a PR |
| Sep 25 to Sep 28 22:53 | First emails killed by the 60 s drain limit; the internal error email then reached the client | 5 first-email kills, 3 held (WS 1, Willows 2); Sep 28 15:02 an internal "could not be composed" error email reached WS's leasing inbox | n/s | a PR (Sep 28 15:42), a PR (22:53) | cried wolf (at the customer: our error alert went to the client) | repo docs/planning/incidents/2026-09-28-first-email-compose- |
| Sep 25 ~05:00 to 09:23 | Post-call save loaded every conversation and hit a 10 s timeout (7 of 54 runs) | 3 real calls lost their follow-up (2 WS, 1 Camellia) for ~4 h; the first report said "no one affected", which was wrong | n/s | ~4 h (a PR at 09:23) | silent (then a false all-clear) | page |
| Sep 25 | Closing 11 old tickets fired a "rate your repair" text for each within 15 s | 8 Camellia tenants got 11 unwanted texts (one got 3 in 5 s); any reply with a digit 1-5 was saved as a rating | n/s | a PR, a PR (Sep 27) | n/s | page |
| Sep 25 14:14 to Sep 27 | Kat (WS staff) texting from an old number was treated as a prospect | WS's leasing agent was run through the prospect flow and offered unbookable times | n/s | a PR, a PR, a PR, a PR (Sep 27) | n/s | page |
| Sep 25 to 26 | A WS voice tour confirmation named the wrong home (unit-number mix-up) | one WS prospect got a tour confirmation naming the wrong house | n/s | a PR, a PR (Sep 26 23:02) | n/s | page |
| Sep 26 01:16 to 13:03 | Upstash Redis hit its 500,000-request cap (a 2-minute keep-alive added Sep 9 exhausted the free tier); the login-budget store failed closed for every customer | robot writes stalled for every client ~11.8 h (inferred) | alert (Sentry only: 14,535 errors) | ~11.8 h; Fede upgraded the plan himself (a PR); robot #430 [urgent] closed unmerged | silent beyond Sentry (no page; Sentry itself dropped 1,325 events that day) | Sentry 1BM |
| by Sep 26 (n/s) | A tour was booked on a home with a pending application | a WS prospect booked to tour a home with a pending application (inferred WS) | n/s | a PR (Sep 26 06:19) | n/s | a PR (PR history) |
| Sep 26 (~24 h) | WS had no weekend tour slots | WS prospects could not book weekend tours for ~24 h | n/s | a PR | n/s | a PR (PR history) |
| Sep 26 to 27 | WS single-family AppFolio showing writes failed | WS single-family homes got no AppFolio showing (count n/s) | n/s | a PR, a PR; robot #432, #433, #452 to #454 | n/s | PR history; robot #432, #433, #452 to #454 |
| Sep 26 to 27 | Outgoing filter redacted digits inside Zoom links; a self-guided home two hours away booked as a staff video tour | a WS prospect got broken Zoom links (404s); another got a video tour for a self-guided home | n/s | a PR (Sep 27 20:01) | n/s | page |
| Sep 26 16:34 to Sep 28 00:51 | Background AI ran on a stale token pool (8 tokens set Aug 17: 3 revoked, 5 capped) | Camellia's concession watcher failed from Sep 27 07:00 (~18 h; effect n/s); test texts went unanswered | a session stumbled on it (weekend audit) | a PR (Sep 28 00:51) | silent (Sentry: 45 fatal events, never resolved; the parity audit said "ok") | page, Sentry 11X |
| by Sep 27 (n/s) | An agent turn timed out (120 s) and ended in silence | a prospect's text got no reply (customer n/s) | n/s | a PR (Sep 27 09:03) | n/s | a PR (PR history) |
| Sep 28 to 29 | Promise scanner pinned a call-center agent's promise on Clara | a first-person note was emailed to WS's leasing inbox ~6 h after the call | n/s | lane turned off Sep 29 | n/s | repo docs/planning/incidents/2026-09-28-promise-misattribute |
| Sep 28 13:16 | A reschedule text did not cancel the 14:00 tour (same house under two property ids) | a WS prospect's old tour stayed booked and its follow-up texts stayed armed | n/s | a PR, a PR | n/s | repo docs/planning/incidents/2026-09-28-cancel-tour-cross-pr |
| Sep 28 ~14:58 to 15:58 | A shared prod leasing prompt was edited, breaking byte-identity | the edit was live ~1 h at Camellia and Yale (effect n/s) | alert (Camellia byte-identity replay job went red) | ~1 h (revert a PR at 15:58) | helped | a PR (PR history) |
| Sep 29 to Oct 13 | CI and Vercel cost ramp | internal only (CI from ~$50 to $275-$752 a day; Vercel from $12 to $43 a day) | n/s | n/s | silent (Vercel never crossed the $100 alert floor) | page |
| Sep 29 ~10:00 | Situs 429 storm on connect (budget + 10 arm rows missing) | AppFolio rate-limited Situs within minutes; sync paused | n/s | ~12 h (rows written 22:22-22:23; a PR automates) | silent (checker blind to paused or unarmed accounts) | session memory |
| Sep 29 | Maps key denied, hidden by a fallback | internal only (dark feature; 0 locations written) | a session stumbled on it | n/s | silent | session memory |
| Sep 29 | Kat's tour alerts never sent (no mobile on her row) | WS leasing agent got no tour alerts (since when: n/s) | a session stumbled on it (turn-on night) | n/s | silent | session memory |
| Sep 29 21:53 to 21:58 | Unapproved JP&Co check-in text switch flip | none (nothing sent; reverted in 5 min) | n/s | 5 min | n/a | session memory |
| Sep 30 10:06 to Oct 1 14:31 | WS Zillow leads silently dropped, then a run of lead-thread bugs | 2 WS Zillow leads got no reply until the fix (10:06, 15:21; Clara re-sent); then mis-threading, nameless placeholder people, follow-ups not reaching the lead, an internal timestamp leaked into a reply | n/s | drop fixes a PR, a PR, a PR; 9 [urgent] PRs Sep 30 19:19 to Oct 1 14:31 (a PR to a PR) | silent (inferred) | standing rule |
| ~Oct 1 | Quoted-chain guard never fired on the Outlook (Graph) lane (a PR regression) | duplicate prospects from Outlook-lane email (count n/s) | n/s | a PR (Oct 1 00:52) | n/s | a PR (PR history) |
| Oct 1 01:33 to 06:14 (page in CT, converted, est.) | A stale worker re-sent ~134 application emails | ~134 application emails re-sent (01:33-02:25 and 04:33-06:14; recipients n/s) | n/s | n/s | n/s | page |
| Oct 1 10:48 to Oct 2 18:06 | Sync fallback wrote one property's residents under another (Oct 1 10:48 and 12:02; Oct 2 11:02) | residents filed under the wrong property or company (company n/s) | n/s | ~31 h (a PR at 18:06 Oct 2) | n/s | a PR (PR history) |
| Oct 1 12:14 | Camellia transfer rang out; Clara said "office closed"; no note saved | a Camellia caller told the office was closed; no message taken | a session stumbled on it (Oct 9 review) | no issue filed | silent (same shape as C020) | session memory |
| Oct 2 to 3 | WS application send failed (robot scope guard) | the failure leaked to the renter and to WS staff; a session proposed asking Kat to do it by hand | n/s | n/s | silent (the refusal never reached Sentry) | page |
| ~Oct 3 | A vacant home in the rent roll was told "no longer available" | a prospect told a vacant home was gone (customer n/s) | n/s | a PR (Oct 3 07:13) | n/s | a PR (PR history) |
| Oct 5 to Oct 13 | A Camellia future tenant's asks went nowhere; callback notices silently refused | a Camellia future tenant's forward (Oct 5-6) reached only an internal alert; Oct 13 14:00 and 14:03 two callback notices were refused (2 of JP&Co's 4 notices since Oct 5) | alert (internal only, Oct 6) | n/s | silent (no Camellia person was told; Sentry logged the refusals) | page, Sentry 1H7 |
| Oct 5 to Oct 6 16:58 | Team notices held as Supervised when the mode was unset | team notices held instead of sent (company n/s) | n/s | a PR (Oct 6 16:58) | n/s | a PR (PR history) |
| Oct 6 01:34 to 01:48 | Clara crashed mid-reply | a customer message got no reply ("nothing was sent to the customer") | alert (Sentry fatal, 12 events) | n/s | n/s | Sentry 1FQ |
| ~Oct 6, Oct 7 and Oct 14 | WS tour writes to AppFolio failed; no guest card | 1 (~Oct 6), 6 (Oct 7) and 5 (Oct 14) failed writes; tours without guest cards (the Oct 14 count likely overlaps C147, inferred) | alert (Smith in #alerts) | program open | helped | page |
| Oct 6 18:43 to 21:12 | Google calendar refresh failed (OAuth client missing in Lambda and worker); a session removed a tour host as "mitigation" | WS tour setup changed by us without asking | alert (18:43) | ~2 h 29 m (Lambda fixed out of band 19:04; worker by a PR and a PR, 20:57 and 21:12) | helped (alert fired; the response was wrong) | session memory |
| Oct 7 ~09:37 to 11:35 | An emergency reply was dropped and logged critical | none known ("resident would get nothing"; caught on the Willows test property) | a session stumbled on it (live Willows retest, 10:39) | ~2 h (revert a PR at 11:35; re-landed a PR to a PR) | helped (the retest caught it) | a PR, a PR, a PR, a PR (PR history) |
| Oct 7 15:28 to ~20:32 | WS robot hit PROBE_UNKNOWN on a two-hop AppFolio login redirect; the guest-card outbox parked after 1 h | ~5 h (est.) of WS robot work failed on the first self-guided tour night; 5 write rows stuck | a session stumbled on it (Oct 8 retro) | ~5 h (est.); robot #479, #480; outbox fixes a PR, a PR (21:12 to 21:30) | late (info level only: 145 "probe ambiguous" events 15:38-20:26; heartbeat and fingerprint-collision gaps) | Sentry 1G5 |
| Oct 7 to 18:59 | A tour reminder text carried a staff-only note as arrival info | a prospect got a staff-only note in a reminder (customer n/s) | n/s | a PR (Oct 7 18:59) | n/s | a PR (PR history) |
| Oct 7 to 8 night | First WS self-guided tour flooded staff | WS staff got 11 escalation emails, the host 10 texts in 5 h (replay: 4 needed); co-signer answer missing from the knowledge base; an Oct 7 17:30 lockbox-code ask was never sent (agent b) | a human noticed (retro) | n/s | cried wolf (customer-facing notices) | session memory |
| to Oct 8 15:14 | Worker KMS permission gap: Gmail refreshes failed silently | n/s (connected Gmail mailboxes could stop refreshing) | n/s | a PR (Oct 8 15:14) | silent | a PR (PR history) |
| ~Oct 8 | 4 WS outreach workflows terminated | zero follow-up touches sent to those 4 WS leads | alert (leasing-workflow-health) | n/s | helped (re-pages; CONFLICT: every 24 h vs every hour; the sweep repairs dropped cards to "closed") | page |
| Oct 8 16:56 to 18:42; Oct 8 to 12 | Sign-in code regression (a PR): codes blocked by the test-property recipient guard; staff codes texted from Camellia Apartments' line | a WS user locked out ~1 h 46 m and complained by email; 5 staff sign-in codes in 4 days came from Camellia Apartments' line (3 after a PR) | a customer told us | a PR at 18:42 (~1 h 46 m; merged without Fede's go); the sender fix needs MFA_SMS_FROM_NUMBER, a Vercel sensitive variable only Gera or Fede can change (a PR is a draft) | silent | page |
| to Oct 9 ~11:34 | Same-day tour offers were never made | 48 of 48 cases (WS 15, Camellia 33); 2 WS prospects corrected at 09:28; the WS backup tour host was never texted | n/s | a PR, a PR, a PR (11:21 to 11:34) | silent (the database held 19 of 19 signals that nobody read) | page |
| Oct 9 15:48 | "Representative?" not transferred | a WS caller asking for a person was not transferred (1 earlier miss, Sep 21 10:14, in ~41 asks over 45 days) | a human noticed (Fede) | n/s | silent | session memory |
| Oct 9 | Emergency calls lost context: the on-call ring had no building or unit; an emergency to a company main line told nobody; a reply was misfiled | a WS water-leak call reached the on-call ring with no building or unit (same call, inferred) | n/s | a PR, a PR, a PR, a PR, a PR, a PR, a PR (08:44 to 22:58) | silent | page |
| Oct 10 09:07 | Flipping the Fairhaven sandbox to Supervised sent per-tenant reviewer emails | internal only (~33-40 emails to the sandbox property alias, bcc hello@; CONFLICT: agent b says WS staff) | n/s | a PR, a PR (10:18 and 10:22) | cried wolf | a PR, a PR PR bodies |
| Oct 10 to Oct 11 ~16:00 | Two-home inquiry got two first emails and two follow-up clocks | a WS prospect got 2 first emails 55 s apart; 2 live follow-up timers ~24 h | n/s | ~1 day (a PR and others) | silent (inferred) | session memory |
| Oct 12 10:40, Oct 13 10:17, Oct 14 10:18 | WS robot hit a daily ~24 h session-expiry redirect loop | internal only (WS robot work paused until a reset that forces MFA) | n/s | robot #514 (Oct 14 10:53) | n/s | robot #514 |
| ~Oct 12 | A resident's door-repair confirmation was cancelled and never sent | a resident got no repair confirmation (company n/s) | n/s | n/s | n/s | page |
| Oct 12 14:25 to Oct 13 13:37 | JP&Co guest-card reads 403 after the robot-user switch; the robot #498 cutover made the universal scope guard fail closed | JP&Co new cards not read ~23 h (inferred); Situs breaker also tripped 7 times from keepalive | a session stumbled on it | ~23.2 h (needed Fede's portfolio grant) | silent | page |
| Oct 12 | A reply that was only an internal note sent nothing; a moving-out reply said "open the email" | a tenant got "open the email" instead of the portal link (customer n/s) | n/s | a PR (16:36), a PR (18:58) | n/s | a PR, a PR (PR history) |
| Oct 13 02:55 to 03:35 | ElevenLabs API degraded | 3 calls carried no audio | a session stumbled on it (overnight build) | vendor recovered 03:35 | silent (no ElevenLabs issue in Sentry) | session memory |
| Oct 13 14:20 to Oct 14 10:20 | Tour slot stuck (calendar write never confirmed) | 1 tour slot unbookable ("nobody else can book these times"); company n/s | alert (Sentry only, 20 fatal) | n/s | n/s | Sentry 1HA |
| Oct 13 13:26 to Oct 14 09:26 | Robot's AI API key rejected (Anthropic 401 on a 175-day-old key, agent b) | 11 AppFolio guest-card writes held up to 20.5 h (2 overnight up to 20.5 h late; 9 that morning up to 1 h 14 min); Situs locked out, JP&Co stayed signed in, WS stayed locked after the key fix; "No message went out because of it"; separately, 3 "add it in AppFolio" emails reached WS staff 09:38-09:46 Oct 14 from a different no-home-tour bug (a PR) | alert (#alerts post 14:03) | 20 h (key swapped 09:26 Oct 14); sign-in back ~20 h 52 m, all writes ~21 h 5 m (agent b); CONFLICT: a page says "19 hours to fix" | late (posted in 37 min but no page, repeat or clock: each ask "was one line among about 30 in #alerts"; overnight retries spent the 24-a-day sign-in allowance; breaker tripped 13:26, 13:58 and 19:58) | page, Sentry 1H4 |
| Apr 29 to Sep 2 | Yale 25 Station had no rent roll; Clara told a prospect there were no one-bedrooms while the website listed some | a Yale prospect was told nothing was available | a human noticed (Yale go-live review) | Sep 2 | silent (no freshness check on the feed; the heartbeat came afterwards) | docs sweep A19; availability-heartbeat.ts:11-13 |
| about Apr to Oct 7 | The website's demo-request form showed "success" and sent nothing | prospective customers' demo requests were lost for six months | n/s | Oct 7 | silent (no check on the form at all) | docs sweep A60; demo-request-silence.detector.ts:5-8 |
| Jul to Sep 23 | Four or five renewals sat in "prepare failed" with no deadline; one resident was texted a price and deadline for an offer that did not exist | a Camellia resident got a wrong renewal text; renewals stalled 64 to 72 days | n/s (the office got a "blocked" notice the next day) | n/s | silent (row nobody watches) | docs sweep A48 |
| Aug 21 | 63 conversations were stuck as "escalated", which keeps Clara quiet for that person; nothing ever expires that state | 63 people got no Clara replies | a session stumbled on it (a production sweep) | n/s | silent (row nobody watches) | docs sweep A12; stale-mute-sweep.ts:5-9 |
| Aug 26 to 31 | Two Camellia leases ended with an unanswered renewal offer still open; the month-to-month switch was skipped every day | two Camellia leases lapsed without the switch | n/s | n/s | silent ("nobody, PM or engineer, was ever told") | docs sweep A24; alert-engineering-lapsed-renewal.ts:13-16 |
| Sep 18 | Nine Western Slope tours were recorded as "reminder sent" when nothing was sent | nine prospects got no tour reminder | n/s | n/s | silent ("no alert fired anywhere") | docs sweep A43; tour-skip-reasons.ts:19-23 |
| Sep 24 | A 7:37 PM tour offer went only to the shared leasing inbox, which nobody reads overnight | a Western Slope prospect's tour offer waited overnight | n/s | n/s | silent (wrong channel) | docs sweep A53; staff-tour-alert-subscribers.ts:4-9 |
| Sep 29 | A large first import parked 233 of 1,135 lead rows before anything tried them | Situs leads parked unreplied | n/s | n/s | silent | docs sweep A56 |
| Oct 5 | SendGrid accepted an approved email and later dropped it; the held message kept saying "sent" | one approved customer email never arrived | a session stumbled on it (a production proof run) | n/s | silent ("nobody heard") | docs sweep A59; bake-gate.test.ts:4365-4366 |
On call means one thing: when Clara cannot work at a customer, a phone makes a noise within minutes and keeps making it until someone says "got it".
Every option starts with the same diet: delete the 80% of #alerts that is plumbing. They differ at 2 AM. Slack alone is not an option: nothing overrides a muted channel.
Delete the plumbing, move the lead alerts out of #agent-trinity, ask Fede to un-mute. No page path.
Deletes: about 820 of 1,027 posts. Adds: nothing. Cost: $0. Fails: yesterday repeats at night.
The diet, plus one "can Clara work here?" check per customer. Thirty minutes of "no" pages both phones, repeats until acknowledged, and silence from the check pages the same way. The page path is Fede's pick:
| Page path | Reaches a phone | Repeats until acked | Second person | Cost | Fit |
|---|---|---|---|---|---|
| B1 PagerDuty Free (recommended) | Calls and SMS, 100 a month; app push (critical-alert override unverified) | Yes (repeat count on Free unverified) | Yes, one escalation policy | $0, 5 users | Native Sentry and SNS intake; one more login; Slack ack not on Free |
| B2 Pushover | Push with critical alert; no calls | Yes, emergency priority, every 30 s for up to 3 h | No escalation; both phones at once | $5 one-time per phone | Simplest: a Lambda posts to one URL; silent if push is off |
| B3 Twilio loop on our account | Call and text from our own number | If we write it | If we write it | About 3 cents a page | No new vendor; a small program we own and must drill |
Deletes: everything in A, plus six Sentry chronic families and the CI capacity alarm posts. Adds: one check, one notifier, no settings. Cost: $0 to $5 once. Fails: only if both phones are off, which the morning clock shows.
incident.io, Better Stack or PagerDuty paid: schedules, rotations, Slack acknowledgement, status pages.
Deletes: everything in A. Adds: a product, two seats, a schedule to maintain. Cost: $20 to $70 a month. Fails: Fede's own rule; a team-of-ten tool with two names on the schedule.
One concern per PR, each mergeable alone and dark, no switch anywhere. Order matters: the diet lands before the page path, so the first page is not drowned.
| # | PR (one concern) | Repo / lane | What it deletes | Proof on the ladder |
|---|---|---|---|---|
| 1 | #alerts diet. CI, deploy, review-bot and cost posts stop posting to #alerts and go to a bot-owned channel (#agent-smith); Willows and Fairhaven never alert. | app + agent-smith + workflows (Fede, Smith) | About 80% of the channel | Unit test with a planted P3 post; one day of replayed posts shows the count |
| 2 | Lead alerts move. "A question has been waiting too long" posts to #alerts, not #agent-trinity; the Trinity flag stays for hard blocks only. | agent-smith (Smith) | A third of #agent-trinity | One real lead alert lands in #alerts |
| 3 | The P1 check. One Lambda every 5 minutes computes "Clara can work at <customer>" from facts that already exist: last robot success, held-write age, line and mailbox pollers, a one-token model call. Writes one state row per customer; posts once per state change with the escalation clock. | app, infra (Fede) | Per-event posts for the symptoms it covers | Planted dead key on Willows flips it red in under 10 minutes and green on recovery |
| 4 | The page path. A P1 state change calls the path Fede picks (B1 SNS to PagerDuty, B2 Pushover URL, B3 Twilio flow), plus a dead man's switch on the check. | infra (Fede); third-party config is Fede's go | Nothing; this is the one addition | A real page to both phones from a Willows drill, acknowledged, and a silence drill |
| 5 | Robot standby key. On "invalid x-api-key" the robot switches to a standby key, keeps working, and raises a P2. The other half, a dead model key never counting as a sign-in failure, already shipped on the robot's main branch (Oct 14, 10:44 AM). | appfolio-browser-agent (Gera's lane; handed over on the bridge thread, waits on this page) | The 24-hour lockout class | Replay of yesterday's log ends with a successful cycle at 1:36 PM |
| 6 | Sentry diet. Resolve or rate-limit the chronic families (budget exhausted, phone coverage, login budget, rent roll); keep one rule: fatal at a customer org goes to the page path. Needs the alerts scope on the token. | Sentry config (Fede's go) | Six families, about 60,000 events | Issue counts the next day |
| 7 | Alarm diet. CI vCPU alarms stop forwarding to Slack; P1-class alarms (inbound queue age, Mac mini heartbeat, replicator flatline) add the page path as a second subscriber. | infra (Fede) | 310 of 390 alarm transitions | A planted ALARM reaches the page path |
Each PR body names which it is (a fix or improvement to live behavior, so no switch) and the rungs it climbed. Nothing here turns anything on at a customer; the page path is a channel to us. PR 5 is a fix in Gera's lane and goes through the bridge thread, never a tag.
Five yes/hold lines, each with a recommendation. "Yes" to all five is Option B.
| Source | What was read | How |
|---|---|---|
| #alerts (C0BGTNSP04C) | 2,843 top-level posts and their threads, Aug 14 to Oct 14; reply users, reactions, Smith verdicts | Slack API with the read token; classified by title rule (CI, cost, CloudWatch, Sentry, evals, test property, Clara at a customer, meta); the top 70 titles checked by hand |
| #agent-trinity (C0BRYAFF16Z) | 392 top-level posts, mentions, Fede's replies and their timing | Same |
| #updates-fede, #engineering-bridge | Incident posts and hand-offs for the ledger | Same |
| Sentry (org propflow-ai, project javascript-nextjs) | 534 issues with first/last seen and counts; received and rejected stats | Issues and stats endpoints; the alert-rules endpoint returned 404 for our token (no alerts scope), so rules are inferred from the Slack posts |
| CloudWatch | 795 alarm history items, 106 alarms, the alerts topic's subscriptions | Read-only describe calls |
| App repo, robot repo, agent-smith | Every notify, alert, cron, workflow and watchdog path, with file and line; all 45 Vercel crons read by hand | Snapshots of main at 0982404d, 067e4db, 8a430ddd; read-only |
| Mac mini | 24 launchd jobs, the Trinity flag events and config, the dead man's switch | Local files; the flag script was read, never run |
| Fede's inbox | Sentry, CloudWatch, GitHub and Vercel mail in 60 days | Gmail search, counts only |
| Prior pages | Sep 12 alert-noise audit, Oct 12 bot-noise page, Oct 14 write-queue RCA | Read; numbers linked, not restated |
| On-call tooling | PagerDuty, incident.io, Opsgenie, Grafana OnCall, Better Stack, Rootly, Zenduty, Pushover, Healthchecks, Cronitor, Slack VIP, Twilio, SNS, Vercel | Vendor pages fetched Oct 14; anything the vendor's page did not confirm is marked unverified |
"Touched by a person" means Fede or Gera replied in the thread or reacted on the post. No prospect, tenant or address appears on this page. Times are Mountain. Nothing was planted in production; the only writes were this page and one Slack post announcing it.
Ledger rules: a row "touched a customer" unless its impact is recorded as internal only, none or not recorded; "an alert fired" means an alert was the detection path; "a person acted" excludes alerts that reached only Sentry or an internal card with no reaction. Six rows from the docs sweep were dropped on the final pass as duplicates or pre-window.