Every phone call to Clara on the shared production line and the Willows test line got cut off about 3 seconds after the caller started talking, for 1 hour 46 minutes. Here is what broke, why our own testing missed it, and what we're doing so it can't happen the same way twice.
2026-09-16 · production, all customers on the shared phone line + Willows bench · Resolved 16:44 MDT
Every inbound call died right after the caller's first sentence, for 1 hour 46 minutes. From 2:58 PM to 4:44 PM Mountain time, anyone who called Clara's shared production line — the line every customer's calls flow through — or the Willows internal test line got about 3 seconds of Clara, then silence. Not one call succeeded during that window.
The trigger was a same-day code change meant to forward maintenance callers to a client's own call center. It shipped fine for the one property that actually has that call-center setting turned on. For every other property — which is every property, because the setting is supposed to stay off until a client asks for it — the change broke the phone system's setup for every kind of call, not just maintenance calls. The fix was a straight revert, live by 4:44 PM. We've since added a guard that would have caught this before it ever reached production.
Of the 19 failed calls: 11 were failed attempts from 2 distinct Camellia callers (one caller retried 5 times, the other 7). Seven were our own Western Slope test-robot calls, and one was a bench-robot call to the Willows line.
Camellia is the only live customer this touched. Western Slope isn't live yet — its public number isn't forwarded to Clara — so no real Western Slope caller could have hit this. Texting and email kept working the whole time; this was a phone-only failure. We checked ElevenLabs' and Anthropic's status pages: both showed all systems normal, so this wasn't a vendor outage — it was ours.
Earlier the same day, a change went live that gives maintenance callers a way to be forwarded straight to a client's own call center, with the right extension digits dialed automatically. That "digits to dial" field is only meant to have a value when a property has actually set up a call-center route — for every other property, dark and empty is the intended, correct state.
The phone system's underlying voice platform (ElevenLabs) doesn't allow that field to be empty — it requires at least one character. Every property without a call-center route sent it an empty value anyway. The platform rejected the phone agent's entire configuration the instant a caller's first sentence needed the model to respond, which is why every call died right after "hello."
The exact, repeatable error from the platform: LLM Cascade Error: 1 validation error for PostDialDigitsStatic — value: String should have at least 1 character (string_too_short, input_value=''), with the call ending as "All LLMs have failed." This is not a guess — it's the literal error returned on every failed call.
The change deployed to both the shared production line and the Willows bench line at 1:36–1:39 PM MDT. The first ordinary phone call after that deploy was at 2:58 PM — nothing had called in between to surface the problem sooner.
The two test calls made before shipping were placed against the one property that already has a call-center route configured — so the "digits" field had a real value on both calls, and both succeeded (and even transferred correctly). Nobody tested a property without that setting, which is the state of every single customer today.
Worse, the automated test written for this change asserted that the digits field must never be empty, and its test data baked in the empty string as the "correct" dark-state value. The test suite was checking that the broken state was right.
Separately, our automated pre-merge check does verify the phone agent's prompts and tool setup render correctly, but it doesn't check that setup against ElevenLabs' own rules for what a valid value looks like — so this class of error has no automatic tripwire today.
We found this ourselves, through a routine Western Slope test call at 2:58 PM — not through any alert. There's currently no monitor watching for ElevenLabs' "All LLMs have failed" error, and none watching for a run of unusually short calls. As a result, real Camellia callers had already been failing for about 50 minutes before anyone on our side knew (first real failure 3:07 PM; the team opened the incident around 3:25 PM).
The fix reverted both same-day changes — the maintenance-forwarding feature and the innocent wording tweak — putting the digits field back to its safe, empty-and-optional state. That revert merged at 4:41 PM and finished rolling out to both phone agents at 4:44:35 PM. Two unrelated process hiccups slowed getting it out (a wording filter false-flagged the ordinary word "transfer," and the fix collided with another change merging at the same time); both were cleared within the same push.
Proof it's fixed: at 4:46 PM, a test call to the Willows bench line — on a property with no call-center route, the exact condition that broke — ran a normal 109-second, 11-turn conversation and answered three leasing questions before the caller hung up. The same agent, on the same kind of property, had died at 19 seconds and 2 turns before the fix. No real customer call had come in yet on the shared line as of this writing; the next one is the real-world confirmation.
| Time (UTC) | Event | Reference |
|---|---|---|
| 19:36–19:39 | Maintenance call-forwarding change (and its wording follow-up) synced to both phone agents | PR #8882 |
| 20:22–20:25 | Unrelated wording change synced; briefly the lead suspect | PR #8903 |
| 20:58 | First failed call (our own Western Slope test robot) — window opens | — |
| 21:07, 21:08, 21:10 | First real Camellia caller failures | — |
| ~21:25 | Incident opened | — |
| 22:15–22:20 | Cluster of 7 failed calls, likely one caller retrying | — |
| 22:37 | Last real Camellia caller failure | — |
| 22:41 | Revert merged | PR #8923 (−926 lines, 20 files) |
| 22:44:35 | Revert finished syncing to both agents — window closes | — |
| 22:46 | Recovery proven: 109s/11-turn bench call succeeds on a no-route property | — |
Window: 20:58–22:44 UTC = 2:58 PM–4:44 PM MDT, 1 hour 46 minutes.
| # | What | Status | Recommendation |
|---|---|---|---|
| 1 | Test every phone-agent change on a property without the new setting first, then with it — never the reverse. | Proposed — pending review | Write this into the repo's phone-config rules so it's not tribal knowledge. Recommend yes. |
| 2 | Build-time guard that fills in every per-call value as empty and fails the build if ElevenLabs would reject any of them empty. | Done — PR #8934, confirmed it catches this exact bug | — |
| 3 | Canary rollout: a phone-config change goes to the Willows test line first, places one automatic test call, and only reaches the shared production line if that call succeeds. | Proposed — pending review | Recommend yes — roughly a day of engineering work, would have caught this before any real caller was affected. |
| 4 | Alerting on the platform's own failure signal ("All LLMs have failed") and on 3 short calls in a row. | Proposed — pending review | Recommend yes — small effort, would have cut the ~50-minute blind window to a page. |
| 5 | When triaging, pull the exact error from the failed call before blaming any specific change. | Done — written into this page as the incident playbook | — |
| 6 | No fleet-wide phone-config change goes out the afternoon before a customer test day unless the canary (item 3) is in place. | Proposed — pending review | Recommend yes. |
| 7 | Deploy skew: the ElevenLabs agent sync finishes about 3 minutes after merge, the Vercel webhook that serves dynamic variables about 13 minutes after. A PR that adds a variable both sides must know is broken for those ~10 minutes (seen 17:33 MDT as error 1008 'Dynamic variable … is not available'). The sync should wait for the health endpoint to report the same commit before pushing agent config. | Proposed — pending review | Recommend yes. |
The same afternoon, a separate mistake sent 100 unwanted emails. At 1:00 PM MDT, Western Slope's own leasing inbox was set as the address for team notices (approved, for tour notices). That also armed an hourly reminder that nobody meant to turn on: "application awaiting review." At 2:00 PM, that reminder fired once for every open application in Western Slope's records — 100 emails, all landing in their own leasing inbox in about a minute. The system's "not live yet" safety switch, which is supposed to keep Clara quiet for customers who haven't gone live, doesn't currently cover this kind of internal team notice — so it went through anyway.
A fix (PR #8926, open) makes these reminders skip any company that isn't live yet, and stops the 100 reminder timers that are still running before they fire again — their next scheduled tick is 2:00 PM MDT tomorrow (September 17), so the fix needs to deploy before then. Whether to also clear the address now or turn it off for this one property in the meantime is Fede's call. The lesson for next time: before pointing any automated notice at a real inbox, list every sender that could use that address and count how many emails each one could trigger.