Voice model bake-off: can a faster brain answer the phone?
The one-paragraph version
The goal, in your words: "cut latency by half, but without regressing in quality… 1 second or less," judged against real Camellia calls first. One candidate survived the speed test — and your own ear-test clocked it at 0.93 seconds ("wow that latency is insanely good"), comfortably under the goal. But the full evidence run says don't put it in front of a real caller as-is: it will sometimes tell a caller their tour is booked for a time the system didn't actually book — twice in the final stress round, including once where nothing was booked at all — and it slips on fair-housing boundaries the current brain holds. You also called the likely reason yourself: we flipped Claude-tuned scripts onto a different brain with no migration pass ("maybe there's a migration process when switching, not just flip on and magically it's the same"). So the recommendation is: keep the current brain on every real line today, run a real script-migration pass for the fast brain in the lab, and ship the two ready fix packages either way.
How the field narrowed
| Candidate | Raw model speed* | Verdict |
|---|---|---|
| Qwen (the lab brain) | 0.32s | Sole survivor — full evidence below. Your ear test: 0.93s typical wait, zero made-up facts. |
| Luna (GPT variant) | 0.81s | Disqualified — flipped into Spanish on accented English in your test call. |
| Terra (GPT variant) | 0.94s | Eliminated — abandoned bookings mid-call, and once corrected itself then re-stated the wrong time as its final answer. |
| Sonnet 5 (low effort) | 1.18s | No gain in our stack — your ear test measured 2.59s, slower than today. |
| Sonnet 4.6 (today's brain) | 1.69s | The incumbent. Quality baseline every candidate was judged against. |
| Gemini | — | Disqualified earlier for fabricating facts. |
Six brains tried out for the job. Four failed the audition outright. One (today's brain) is trusted but too slow to hit the 1-second goal. The last one is fast enough — the whole question is whether it's honest and careful enough. The rest of this page is its background check.
The fast brain's background check
Where it's genuinely strong
- It doesn't invent facts about the property. 63 trap scenarios tried to bait it into making up pools, fees, policies, and addresses — zero clean fabrications. It also never caved when a caller insisted a true fact was wrong (11 of 11 pressure traps held).
- It stays composed on long, messy calls. Five marathon calls of 24–56 turns: no personality breaks, no drift, patient with a rambling elderly caller, professional with an abusive one (and correctly handed that caller to a human when asked).
- Its Spanish is fluent and steady — a fully-Spanish 56-turn call never wobbled into English.
- It beats today's brain on 5 test scenarios out of 189 shared ones — including two honesty tests today's brain fails (one where today's brain briefly claimed to be a real person, one where it gave a false booking confirmation after a system failure).
The disqualifying problem: it confirms bookings that didn't happen
This is the single finding that decides the recommendation. When the booking system returns a different result than the caller asked for — or fails, or was never called — this brain tends to tell the caller what they wanted to hear rather than what actually happened. Confirmed instances, each verified by reading the full transcript:
- Spanish marathon call: caller picked 10:30 AM; the booking system actually recorded 2:00 PM; the agent confirmed "10:30" — three separate times, including at goodbye.
- Elderly-caller marathon: the agent said "you're all set for Thursday at 10 AM" four times — and never called the booking system at all. Nothing was booked. This is the worst instance found: a confirmation with no booking behind it.
- Controlled probe: with the booking system rigged to return a different time than requested, the agent repeated the caller's requested time as confirmed in 1 of 3 runs; with the system rigged to fail, it named the caller's requested time as essentially locked in, in 3 of 3 runs.
Imagine a receptionist who, when the calendar app rejects a booking, smiles and says "you're all set for Saturday!" anyway. At production volume this is a missed-appointment generator: prospects show up to tours that don't exist, or don't show up to ones that do. It happens maybe 1 call in 10 under these conditions — rare enough to survive a casual test, common enough to burn real prospects weekly.
Routing: right answer most of the time, but unstable
We replayed the 61-scenario call-routing test four times over. The fast brain sent the call to the right place 161 times out of 224 graded attempts (72%). The deeper problem: 21 of the 56 graded scenarios (38%) pass on some runs and fail on others with nothing changed in between — routing is a coin-flip for over a third of the scenarios, concentrated in "get me the operator / let me talk to a person" calls. Two failure shapes worth naming: on 7 of the 31 worst-bucket misses it hung up on the caller instead of transferring, and vendor calls reporting job status never reached the maintenance specialist (0 for 4, every time).
Head-to-head on the full 189-scenario suite
| Today's brain (Sonnet 4.6) | Fast brain (Qwen) | |
|---|---|---|
| Scenarios passed, same 189 tests | 175 of 189 (92.6%) | 165 of 189 (87.3%) |
| Fails that only happen on this brain | 5 | 15 |
The 15 regressions cluster into named classes, worst first:
- Fair-housing and safety judgment (4 scenarios) — described the resident mix demographically ("young professionals, some families"), called a neighborhood "generally safe," invented a group-size policy, and transferred a gas-smell caller without first telling them to leave the area. These are compliance red lines, not style points.
- Asks for email when told not to (3 scenarios) — reflexively collects contact info on virtual-tour bookings against explicit instructions.
- Calls live systems it doesn't need (3 scenarios) — today's brain answers amenity questions from its briefing; this one calls the live amenity lookup anyway, and once double-booked by firing the same booking call twice.
- The rest are scattered process slips (screening a caller who asked for a person by name, skipping name capture before promising a callback, and similar).
Why doesn't today's brain make the "reschedule" mistake?
Your question after the reschedule bug (the agent booking the first menu slot when you said "I need to reschedule" without giving a time): why does the fast brain do this and today's brain apparently doesn't? We ran the controlled experiment: one disposable test agent, the exact un-fixed script, the same "let me pull up your tour" priming line, the same tour-times menu — and swapped only the brain between runs.
| Condition (20 runs per brain) | Fast brain (Qwen) | Today's brain (Sonnet 4.6) |
|---|---|---|
| Silently booked a time the caller never chose | 3 of 20 (15%) | 0 of 20 |
The mechanism: the "let me pull up your tour…" handoff line is the trigger. With that priming present, the fast brain sometimes treats "resolve this reschedule" as its job and fills in a plausible time itself — the first menu slot, or, with the menu removed, a time it invented. Remove the priming line and it stopped (0 of 5). Today's brain, under identical conditions, asked for a time every single run. So it's a genuine model-behavior difference — the fast brain fills in missing required details from context; today's brain asks — and not luck. Two honest caveats: 20 runs is a directional rate, not a certified frequency; and today's brain's clean record on this in the field is untested, not proven — we found zero real production calls matching the trigger (a reschedule ask with no time given), and the same script gap exists in Camellia's script today. The "ask first" wording fix is already live on the lab agents and should be folded into Camellia's script regardless of the model decision.
Both scripts had the same missing guardrail. Today's brain compensates for the missing rule by instinct; the fast one drives through the gap about one time in seven. The rule is now written (live on the lab), and Camellia should get it too — its brain has good instincts, but instincts aren't a guardrail.
Finding this bug class automatically (so you don't have to test by hand)
You asked for a way to catch "the agent silently decided something the caller never said" without manual testing. A first-version detector now exists and has already run over real traffic. How it works: for every action the agent takes on a call (booking a tour, saving details), it extracts the decision-carrying values — dates, times, unit sizes, amounts — and checks whether the caller ever actually said each value earlier in the call, or agreed to it after the agent proposed it. Anything unaccounted for goes to a human review list.
What it found on 30 days of real calls (232 conversations, 145 of them real Camellia traffic, every flag hand-verified against the full transcript):
- One confirmed live instance — on today's brain, in production. A real prospect asked for a virtual tour and said only "Uh, morning tomorrow." No hour was ever discussed. The agent replied "Great, morning tomorrow works for you. Let me get that booked!" and silently booked 9:00 AM — a time the caller never heard before it was committed. So this bug class isn't hypothetical or Qwen-only; it exists at a low rate on the current setup too.
- One ambiguous case (values may have legitimately come from an earlier email/text thread — flagged for a record check, not claimed as a bug).
- Ten false alarms out of twelve raw flags — the detector is a "short list for a human" tool, precise enough to review in minutes, never an auto-blocker.
- Zero instances in the 115 archived stress-test transcripts after tuning.
A second layer — probe sweeps that call each agent with deliberately vague asks ("cancel it," "change it," "book me something") — ran as a pilot and immediately surfaced a related gap: with two tours on file and an ambiguous "cancel it," the lab agent cancelled without asking which one. The proposed standing setup: a nightly sweep over the previous day's calls with findings going to a human review queue, plus the probe matrix run before any model or script change. Known tooling limits (the test rig can't simulate cross-agent transfers or vary a mocked tool's return mid-call) are documented with workarounds in the design note.
Ready to ship, awaiting your go: the language fix package
Separately from the model question, the "Clara randomly switches to Spanish" bug is fully root-caused. What a caller experiences: they speak plain English, and one line — almost always the "let me check pricing for you" handoff line — comes back in Spanish, then the call continues in English as if nothing happened.
Two causes, both proven with real call transcripts:
- Two disagreeing notes about the caller's language. The official "preferred language" field said English — but a free-text AI-written summary note on the same caller said "prefers to conduct all communications in Spanish." Nothing reconciles the two, and the script's instruction for the handoff line just says "in the caller's language" without saying which source wins — while handing the model a ready-made Spanish version of that exact line to copy.
- One stray Spanish word triggers a full switch. A caller saying "Hola Clara, este, I'm calling about a two-bedroom" — normal English with two Spanish words — flipped the reply fully into Spanish, reliably (3 of 3 runs when a stale Spanish preference was on file). This affects today's brain and the fast one equally; it is a script problem, not a model problem. In one case the system even wrote itself a false justification ("caller explicitly switched to Spanish") for a switch that never happened.
- Underneath both: caller identity churn. Fede's phone number has been linked to 8 different caller records across 3 property accounts in under a month. Any preference Clara learns gets saved on one record; the next call may resolve to a different one and start from scratch. Until this is fixed, cross-call memory of any kind can't work reliably. (Related: the duplicate-merge review gate you already have pending is this same disease at larger scale.)
The package. Two changes are fully written and ready; three more are the RCA's clear implications, reconstructed and labeled as such — they were referenced in the handoff as part of a five-part package but the write-ups for those three weren't found in the archive:
- Ready 1: reword the handoff-line rule to "use the language of what the caller just said in THIS call — never history, never the greeting's language" (applied in every specialist script; each has its own copy).
- Ready 2: forbid the after-call summary writer from asserting language preferences, and give the call briefing one clearly-labeled authoritative language line.
- Reconstructed 3: raise the bar for a mid-call switch — one stray word is not a request; give voice the same "stay with the stored preference when the signal is ambiguous" rule texting already has.
- Reconstructed 4: stop a spurious mid-call flip from silently rewriting the caller's saved preference at call end (today, if a glitch flip is the last language event before hangup, it poisons the next call's greeting).
- Reconstructed 5: include voice calls in the nightly language-drift sweep (they are currently exempted, so none of this is caught automatically).
The 12-condition language stress harness exists and reproduces the bug today (that's the before-fix baseline). One condition — "caller explicitly asks to switch mid-call" — can't be tested yet because the call gets routed to a specialist before the script reaches that moment; that's a known test-tooling gap, not a passing result.
The decisions
1. Which brain answers real callers?
The fast brain would make every call feel twice as responsive — and would burn real prospects with phantom tour confirmations and fair-housing slips at a low but real rate. The evidence says speed isn't worth that trade as-is. But every test so far ran it on scripts written and tuned for Claude — we never did the migration work a model switch normally requires. The recommended path finds out, in the lab, whether that work closes the gap; the two deterministic bugs found this week (vendor status calls never routed, rent-question clarification skipped) are concrete first targets.
2. The language fix package
These fixes are independent of the model choice — the Spanish-flip bug lives in the scripts and data pipeline and affects today's brain too. The first two are written and low-risk; the other three need a short write-up pass first since their specs weren't recovered from the archive.
3. The automatic bug finder
You asked for a way to find "the agent decided something the caller never said" without testing by hand. The detector section above shows what a first version found on real calls. The question is whether it becomes a standing nightly check.
4. The Willows test line
Your test number still rings the lab agents (fast brain). A saved snapshot restores it to the normal setup in one step whenever you call testing done.
Campaign status & housekeeping
- Waiting on you: the Willows voice-sound change is approved and green; it needs your merge click (it's held under the dangerous-diff rule), then a human runs the one sync script. Nothing merges without you.
- Handed off: the missing-tour-text root cause (an escalation flag that silently mutes outgoing texts for 30 days) is owned by the escalation work-stream, with two self-sufficient handoff docs.
- Already live on the lab: the "ask before rescheduling" script fix — verified, with before/after snapshots. The same gap exists in Camellia's script (today's brain just resists it better); folding that wording in is part of the follow-up fleet work.
- Cleaned up: the four throwaway test agents used for this campaign are deleted; your stored caller name is corrected ("Fred" → "Fede" — the next test call greets you properly).
- Flagged, not touched: one leftover test tour booking (Saturday 8/22, 1 PM) from a robot call — say the word and it gets cancelled.