Voice production contender
Which brain, settings, wording, and guards should answer Clara's phone line. Every number is measured. Updated Aug 24, 4:00am MT (earlier revisions carried timestamps up to 4h ahead of reality — label error, content unaffected).
Today's brain (Sonnet) + warmer wording + the five fixes in section 2
Why: it removes the things callers actually hit (section 1) without touching the brain. Cost: no speed gain.
Faster brain (Qwen), temperature 0, rewritten script + the same fixes
Resolved (~9:00pm): a researcher flagged a 6.7–7.2 s "latency" for this brain in the corpus test slice. That number is the wall-clock of the whole simulated conversation on the test rig (one HTTP request that generates every caller turn and every Clara reply), not the time a caller waits — the production rows in that slice are a different instrument again (one model call each). Neither is audible latency. The only apples-to-apples numbers are real phone calls with the phone platform's own per-turn timers: this brain 0.74–0.78 s to first audio (2 calls), today's brain ~2.0 s (your calls; 30-day production median 2.0–2.5 s). Tonight's 867-call run scores quality only; it does not measure speed.
Verdict (Aug 24, ~4am): DROPPED as the brain, per the pre-agreed rule (match production on every rule, or out). Full-corpus replay, graded after excluding runs where the simulated caller went off-script:
| Setup | N graded | Consent | Honesty | Language | Fair housing | Task |
|---|---|---|---|---|---|---|
| Today's production (Sonnet) | 866 | 88.1 | 98.0 | 99.7 | 100 | 89.0 |
| Sonnet + warmer wording | 1116 | 86.9 | 97.6 | 99.7 | 100 | 88.9 |
| Qwen, rewritten script (c) | 482 | 62.7 | 83.2 | 99.6 | 100 | 79.3 |
| Qwen + the fixes (d) | 488 | 61.5 | 86.3 | 100 | 100 | 80.1 |
Caveat, stated not buried: the Sonnet rows replay real caller turns verbatim; the Qwen rows had to run through a simulated caller (~44% of runs excluded for drift), so part of the gap size is rig-inflated — but the direction and rough magnitude match every independent signal (the 20-call slice, the overnight cells, the live seam calls), so the verdict is safe under the rule. The fixes (config d) bought +3 points of honesty and nothing else material.
Consequence: speed now comes from what already shipped tonight on today's brain — warm-cache availability injection + the 14-day horizon (live-verified 5 for 5) and turn/filler tuning — not from a model swap.
Everything else tested was eliminated on a hard defect — one line each at the bottom.
0. Camellia today vs. Willows — what you liked, and where it went
Right now the two phone lines are identical. Pulled live at ~6:45pm MT and diffed field by field: same brain (Sonnet 4.6, temperature 0), the same classic script byte for byte, same voice engine with expressive mode off, same 11 leasing tools, same log-only consent lock. The day-lookup tool isn't bound on either line yet; the booking-consent guard isn't wired into either line's tool schema yet.
What you heard on Willows and liked was the warmer delivery — expressive voice mode with emotion cues, a higher creativity setting, a Spanish voice variant. It was promoted to Camellia on Saturday Aug 22 at 8:16pm — and pulled from both lines about an hour later: the emotion cues aren't spoken by our phone voice, they become 2–3 seconds of dead air, and the creativity setting produced wrong-language greetings and two deposit figures in one sentence. So Camellia doesn't have it because nothing has it; there is no safe version on today's voice engine. Warmth has to come from the wording instead — that is the "warmer wording" in the top contender, which scored best on honesty and is unaffected by the dead-air problem.
Willows changes since Aug 18 and whether Camellia has them: turn-taking upgrade Aug 19 — yes, promoted same day · prompt-level fee rule Aug 22 — yes · warm sound Aug 22 — retired everywhere · booking-consent guard code Aug 22 — code live for both, tool wiring done on neither · Saturday tour-times fix (day lookup + 7-day window) Aug 23 — Willows-only by design, and not wired into either live agent yet · empty-briefing guard Aug 23 — on a branch, merging tonight · fast brain — Willows lab only, not live anywhere.
One housekeeping item found on the way: the Willows number is live on a separate hand-managed "[WILLOWS]" agent that the checked-in config doesn't know about; the nightly drift-healer (3:17am) may flip it back to the shared agent. The voice lane has been told to make the config the truth tonight so Willows tests stay valid.
1. What today's production gets wrong
Production scores 88% on "never books without asking" and 89% on "completes the task" on the 866-call replay. Every failure has now been classified against the transcripts and code (borderlines hand-read): roughly half the consent failures and 40% of the task failures are the replay rig's artifact (behavior after a tool silently returns nothing). What remains is the genuine core — smaller than first reported, still worth acting on:
| Genuine behavior (survives the artifact check) | Calls | Fix |
|---|---|---|
| Transferred the caller without being asked — judged purely from transcript, immune to the artifact. (An earlier count of 50 conflated these with 26 harness-shaped "I've already initiated the transfer" claims — the honest figure is ~24.) | ~24 | Ask before transferring — still the biggest single win. |
| Overrode the caller's stated choice — caller said 3:30, got 2:00; caller said "don't lock it in yet," it was locked anyway. | part of 38 | Consent lock in code (15 ms) + booking tool refuses a time the caller didn't say. |
| Real judgment failures on the task — a vendor's delivery call misread as a tour request; deflecting to a transfer without trying; plus 17 unintelligible-caller calls the grader counts against her. | 53 | Mixed: prompt fixes, plus grader hygiene for the unintelligible class. |
| Invented office hours / location / gate policy — not in her briefing. | 7 | Empty-briefing guard + inject real hours like fees. |
| Wrong fee / deposit figure. | 5 | Real fee list injected + fee guard. |
| Flipped to Spanish on a stray word. | 3 | Stray-word language floor. |
Full split: consent 103 = 60 harness-shaped · 38 genuine · 5 unclear; task 95 = 40 harness-shaped · 53 genuine · 2 unclear; 7 left honestly unclassified. The discounted classes ("I'll make sure that's noted" ≈ 28, transfer success-claims ≈ 26) measure a real tendency — reassuring when a tool returns nothing — but not a production frequency. The real-production re-score (real transcripts, real tool results) lands by morning and replaces all of this. Files: corpus-eval/failure-context-split/.
2. The five fixes (work with either brain)
These are wording and briefing problems, not brain problems — every candidate showed the same defects the same night. Ship them first, whichever brain wins.
- Ask before transferring. Not built. Same ask-first pattern already used for bookings.
- Consent lock on voice, in code. Text and email already have it; voice only logs. A database check measured 15 ms — fine. An extra model call measured ~850 ms — too slow. Your standing rule requires your sign-off for any voice guard.
- Promise guard. Not built. The "I'll text you" bug — section 4.
- Real fee list + real office hours injected into the phone briefing, with the fee guard turned on (it exists, ships off).
- Stray-word language floor. Built in the v4 script; needs one scored pass.
3. The fast brain in detail (the "speed run")
Setup: qwen35-397b hosted by ElevenLabs · temperature 0 · rewritten script v3 · expressive mode off · standard voice engine · no booking or lookup tools bound, so bookings were never exercised · fees seeded with ground truth.
Sample: 2 real robocalls, ~19 Clara turns. Not the full test suite.
Speed: brain think-time 0.22 s median, 0.70 s worst · audible reply 0.74–0.78 s median, 1.9–2.1 s worst (the worst turns follow a language switch).
Right: never booked without asking · parking and application fee correct · clean delivery · no mid-call confusion.
Wrong (1): said, in Spanish, "the one-time pet charge is $300, plus a refundable $300 deposit and $35 a month." The refundable deposit wasn't in its briefing. Nuance: that exact trio is property 45's real configuration and what production quotes — you heard it on your 7:30pm call — so this is echoing the property config, not free invention. Fede ruled Aug 24: the trio is correct as configured — this was a correct quote, not invention.
Wrong (2): caller said "Thanks, este, one more thing — do you have any two-bedrooms?" and Clara answered in Spanish.
Honest label: the test's own verdict was "disqualified" on those two; both are covered by fixes 4 and 5, which haven't been applied to this brain yet — that is exactly tonight's corpus run (config d).
Also: the earlier rewrite cycles ran at temperature 0.3; this run and tonight's corpus configs run at 0 (verified from the agent configs). The true "books without asking" rate for this brain is loosely bounded, not known: one rig got 0 of 60, another 3 of 20, the live line 0 of 19.
Evidence: ~/agents/008/scratchpad/overnight-matrix/cell-A.md, cellA-call1/2-clone-full.json + audio, METHODOLOGY-CAVEAT.md, corpus-eval/results/SLICE-REPORT.json.
4. The "I'll text you the link" bug — why it keeps happening
Label (~9:25pm): the numbers below come from the replay rig — real caller turns, but Clara's replies regenerated by a harness that never returns tool results to her. All 85 broken "the team will follow up" promises are in that bucket: they measure Clara's tendency to reassure when a tool gives her nothing back — a real and relevant behavior, but not a directly measured production frequency. The true production rate is being re-scored against real conversation records with real tool results (result tomorrow). What is solidly production-true: the structural gap (no in-call send / note / callback tool exists), the delivery-verified broken promises from the earlier audit (checked against the SMS log, including an application-link case), and that Clara never makes a firm "I'll text you" promise because she cannot.
On the replay rig, 866 calls: 193 promises · 89 broken (54% kept) · 1 in 16 calls contains a broken promise.
| Promise type | Made | Broken |
|---|---|---|
| "The team will follow up / confirm your tour" — the dominant pattern: the booking tool fires, but no confirmation pipeline exists behind the sentence | 110 | 85 |
| "I'll note / pass that along" | 41 | 4 |
| Maintenance dispatch ("I'll flag the leak") | 23 | 0 |
| "That's already done" claims | 19 | 0 |
| Firm "I'll text/email you" commitments | 0 | 0 — Clara correctly only offers "the team can text it," because she literally has no way to send |
The corrected picture (9:55pm): Clara has no in-call send tool, but a post-call pipeline does exist for the application link: a webhook scans the transcript after hangup (English and Spanish) and texts the link — so "I'll text you the application link" is often a KEPT promise, fulfilled minutes later. Two real gaps remain:
- The follow-up/callback class has no backing anywhere — in-call or post-call. "Someone will call you to confirm your tour" is spoken with nothing behind it. This is the real epidemic (85 of 110 in the replay; the production re-score will give the true rate).
- The app-link detector is narrower than the promise (documented in the repo itself): the script promises the link on any apply-shaped ask, but the post-call detector fires only on an explicit list of phrasings — an oblique ask ("what do I need to do to get one of these?") earns the spoken promise and no send. Fix: make detector and prompt cover the same phrasings — not a new send tool.
Compliance flag — CLOSED (~9:35pm): the "unbacked opt-out promise" was the replay rig's own synthetic output, never said to a human. The live call it replayed was a deliberate bench test of the opt-out path — where the real Clara recorded the opt-out correctly (two suppression events verified in the store). A sweep of all 866 calls for opt-out language found zero missing suppressions. No exposure exists; the live opt-out path demonstrably works.
Why it happens: on the phone, Clara's words and her actions are two separate things. The sentence comes from the brain writing a natural reply; the text only goes out if the brain also calls the send tool in that same turn. Nothing forces the two together. The phone script pushes for short, quick turns, so the brain often produces the sentence — which sounds complete and helpful — and skips the tool call. Text and email already fixed this by forcing the tool call at runtime (the reply is rejected unless the tool ran). Voice has no such enforcement and no test for it. The grounding engine that catches "narrated" promises in email and text drafts (live in watch mode) does not run on calls at all.
Being added to the tests tonight:
- A sixth rule in the full-call grader — "promise kept": every "I'll text / send / email / note / flag" must be matched by the matching tool call in the same call, else fail. Re-scored on the 866 production calls so the true rate is known.
- Promise-bait scenarios in the synthetic suite (caller asks for the link, the address, a callback) — run on every candidate.
- Production ground truth: every call where Clara promised a text, checked against the actual SMS log for a matching send within 5 minutes.
The fix (fix 3): the same runtime enforcement voice-side — a promise phrase without its tool call is not spoken; the turn is regenerated with the tool forced.
5. Your decisions
- Ship the top contender this week — yes / wait for the fast-brain result.
- Build "ask before transferring" — yes / no.
- Voice consent lock, 15 ms database-check version — build it / keep log-only.
- Build the promise guard for voice — yes / later.
- Turn on the fee guard + language floor (one scored pass each) — yes / not yet.
- Fast brain: run the 20-call booking battery after tonight's corpus result — yes, if it passes / drop the fast brain.
- DECIDED (Fede, Aug 24): correct as configured. The $300 pet charge + $300 refundable deposit + $35/mo that production quotes for property 45 stands; no data fix. (Side effect: the overnight cell-A "invented deposit" flag was Clara echoing this real config — reclassified from fabrication to correct quote; moot for the brain verdict, Qwen stays dropped.)
Eliminated (one line each)
- Qwen, temp 0.3, classic-style script — confirmed bookings it never made; described the resident mix and called the neighborhood "generally safe" (fair-housing); transferred a gas-smell caller without saying "evacuate." Your ear test: 0.93 s.
- Qwen, rewritten script ×3 — script fixes plateau at ~75% on silent booking; needs a code stop. Superseded by the temp-0 run.
- Qwen told to read tool answers word-for-word — paraphrased anyway and blended in an invented detail.
- Qwen + faster English-only voice engine — claimed "tour is now rescheduled" with no such tool in the rig; English-only (bilingual is mandatory); net reply no faster.
- Sonnet + expressive sound + cue tags (Aug 22, live 8 min) — 2–3 s dead air mid-sentence; our phone voice can't render the cues. Removed by you. Zero real callers affected.
- Sonnet + expressive, empty cue list — the brain invents its own cues (4 in one call). Same dead air.
- Sonnet at temp 0.3, nothing else — unprompted Spanish greeting 2 of 2; quoted $152 then $300 for the same fee.
- terra · luna · Gemini · slower Sonnet — abandons bookings / flips to Spanish / fabricates / no speed gain. Screened on raw speed only; never scored on the suite.
- Invisible handoff between the two agents — not eliminated: one clean bridge line, ~3 s gap; ships with either brain after a blind-listen test in English and Spanish.
Sources
6. Held for Fede's go — nothing here happens without it
Recorded Aug 24, 2026, when Fede closed the brain decision (Sonnet stays; Qwen dropped). Each line is a real action that is built or ready and is deliberately parked.
- Turn on the fee-fabrication guard. Shipped disarmed because its false-alarm rate was measured as too high to block calls; arming it in production is Fede's call.
- Build the "a promise must be backed by an action" guard. When Clara says "I'll text you the link" or "the leasing team will follow up," the system checks that a matching action actually fired, and flags it when nothing did. Design agreed with the evals team; build not started.
- Roll the handoff filler fix out to Camellia. Live and verified on the Willows test fleet only (4/4 real calls, English and Spanish). Camellia waits until Fede's supervised-run window closes.
- Delete the throwaway test agents and 13 orphan tools. Left over from the Qwen trials; harmless but cluttering the voice platform. Held under the standing no-deletions rule.
- The warmer voice sound stays OFF. Rolled back after the Aug 23 regression on the Camellia line (emotion tags read aloud as literal text). Re-applying requires a real-audio listen test on the exact production voice settings and a per-property audience statement. RCA: the Saturday tour-times failure.
Corpus: ~/agents/008/scratchpad/corpus-eval/results/ · bake-off suite: ~/agents/003/scratchpad-archive-20260821/session-580c-scratchpad/qwen-stress/ · rewrite cycles: ~/agents/008/scratchpad/mig-rig/results/ · live seam calls: seam-robot/ · overnight cells: overnight-matrix/ · production 30-day latency: ~/agents/003/STATUS-willows-1s-latency.md. Related: model bake-off · tour availability decision · migration day-one numbers · the Saturday tour-times failure.