Proposed — pending review · 2026-08-13 · answers Sean's question "how can we make the voice agent more human" · sources: 20 recent live Camellia calls + the 41-call recorded corpus (404 agent turns analyzed), live ElevenLabs configs for all 11 agents, prompt audit, ElevenLabs platform research

Make Clara sound human

What actually reads as "robot" on our real calls, why it happens, and the ranked fixes — most are one config field we never turned on.

The goal

A caller should feel they're talking with a person, not taking turns at a machine. Benchmark named by Fede: ChatGPT-Voice-class interaction — quick responses, graceful interruptions, no awkward stops, never the same recording twice. Every recommendation below is graded against that bar, and "more human" becomes a number: a naturalness harness scores it call over call.

What callers actually hear (measured on real calls)

2.1s median

pause before every reply; 40% of turns over 3s; worst real case 8.9s of silence then "Sure!"

15 / 20

calls open with the character-for-character identical greeting; "Got it — one sec." verbatim in 35 of 41 walk calls

2–7×

Clara's words vs the caller's, per call; longest single turn 88 words (~35s of unbroken talking)

0 / 404

agent turns with any human reaction — no empathy beat, no rapport, ever, in either call set

5 / 20

calls where the last thing the caller hears is the single word "Sure!" before the transfer ring

51%

of agent-to-agent transfers open with a redundant second acknowledgment (separate fix already carded)

Caller confirms their email — "That's correct. That's correct." — then 13.3 seconds of silence, then: "Perfect!"

Why — five causes, all fixable

The expressive-speech switch is off exactly where it matters — and nobody chose that. All agents run ElevenLabs' expressive conversational voice model, but expressive_mode — the switch that delivers emotion, emphasis, and natural reactions instead of flat reading — is off on Triage, Leasing, and Maintenance (the three agents callers actually reach) and on for three agents almost nobody calls. No PropFlow code ever wrote this; it's unmanaged dashboard state. Related: the "suggested audio tags" system (how a voice sounds apologetic to an upset tenant vs upbeat on good news) supports 20 tags per agent — we define zero.
Clara has no personality on the phone. The shared personality block — literally headed "THIS DEFINES HOW YOU SOUND ON EVERY CHANNEL" — is imported by text, SMS, and email… and by zero of the 11 voice prompts. Voice prompts spend 1.5–4% of their words on how Clara talks; the one lane that was rebuilt against live founder ear-tests (vendor calling) spends 13.2% and sounds it. The main agents open with a bare role label and 12,000 words of procedure. Some prompts mandate the exclamations the personality file bans ("Absolutely!" / "Of course!"), prescribe one fixed line per situation ("Let me check pricing and availability for you" — said even to a caller who asked for a person), and order three agents to close every action with the identical "Anything else I can help with?" — a tic that was already diagnosed and fixed on Leasing, but never carried across.
Temperature zero: the same sentences, byte for byte, forever. Seven inbound agents run their language model at temperature 0 — greedy decoding, so identical context produces identical phrasing on every call. (Our outbound agents run 0.3–0.85; the fleet is internally inconsistent with no code governing it.) Anyone who calls twice hears the same recording.
Turn-taking is a generation behind — this is the ChatGPT-Voice gap. All 11 agents run ElevenLabs' old turn model (turn_v2). The current default (turn_v3) reads speech prosody to tell a real interruption from an "mm-hmm" backchannel — exactly the "interrupts you / awkwardly stops talking" complaint. We hand-typed a 4-word ignore list on 2 agents to paper over it; 8 of 11 agents have zero interrupt protection (a tenant's "mm-hmm" cuts Clara off mid-sentence), ElevenLabs' curated default list is actively suppressed everywhere, and there's no Spanish backchannel coverage despite live Spanish support. Measured: 16% of speaker changes in the walk corpus overlap.
Dead-studio silence. Ambient room tone is off on every agent — the total silence between words reads as "recording." (Landmine: the volume is pre-armed at 4× ElevenLabs' recommended level with loop-pop protection off, so whoever flips it on in the dashboard ships a call-center roar.) Also: two outbound agents — the ones that call tenants unprompted, our highest-stakes robot-or-human moment — were left behind on the old, least-expressive voice model entirely.

Structural root cause, same as the architecture doc: not one voice/TTS/personality dial is owned by code. The entire humanness surface is dashboard-only, unreviewed, and unguarded — which is precisely how the expressive switch ended up off on the busiest agents. Fix is the third instance of an existing pattern (turn and language sync scripts already work this way).

The fix, ranked

#	Change	Effort	Why this order
1	Expressive mode ON for Triage/Leasing/Maintenance + define a small warm tag set (reassuring, apologetic, softly…)	~half a day + branch A/B	Highest single-change impact; we already pay for the expressive model and get none of it
2	TTS under config-as-code (new sync script owning voice, model, expressive mode, tags, temperature — modeled on the existing turn sync)	1–2 days	Makes every other change reviewable/revertible; ends the unmanaged-dashboard era
3	Turn model v2 → v3 + inherit ElevenLabs' backchannel defaults + Spanish coverage, fleet-wide	~half a day + real-call regression pass	Directly closes the ChatGPT-Voice interruption gap; fixes 8 agents with zero interrupt protection
4	Ambient office tone at the correct low volume	one afternoon of listening	One field; kills the dead-studio silence; instantly reversible
5	Temperature 0 → 0.3 on Leasing + Triage, eval-gated	one field + eval run	Ends byte-identical calls; the one config change that could move answer accuracy, so it rides the eval suite
6	Prompt naturalness pass: give voice the personality block; 3 alternates per hand-off line instead of 1; carry Leasing's "vary the closer" fix to the other three agents; turn verbatim-mandated sentences into content mandates; let Triage react in three words before routing; mirror the caller's turn length	~2–3 days + judged evals	The taste work — sequenced after the switches so it lands on a voice that can express it
7	Upgrade the two left-behind outbound agents to the expressive model	~1 day	They call tenants unprompted on the flattest model in the fleet
8	Naturalness harness: inbound warmth scenarios (judged: "reacted to what they said", "closer wasn't canned", "no verbatim repeats") + nightly naturalness score on real calls	~2 days	Makes "more human" a number Sean watches move, and stops regression

Also queued from this audit: switch spoken-number formatting to ElevenLabs' post-generation normalizer (cleaner transcripts, frees prompt budget), retire the deprecated latency parameter, and cap spell-back confirmation loops at two attempts before switching channels (one real call burned 105 of its 303 seconds on a single email address). The 2–3s per-turn pause and the transfer double-acknowledgment are already being fixed (Aug 13 fleet + carded seam fix). Every change ships through PR + sync — never dashboard edits — and is A/B'd on a branch copy with the simulation gate before touching live calls.

Decisions

Answers save on this page and are shared.

H1 — The switch package (expressive mode + turn v3 + ambient tone), branch-tested then live?

Go — ship the package behind a branch A/B and the sim gateRECOMMENDED
Three config fields, all reversible, all validated on a branch copy before live calls. One change at a time, a week apart
Cleaner attribution, slower to the goal. Hold — discuss on recordings first

H2 — Temperature off zero?

Yes — 0.3 on Leasing + Triage, gated on the eval suites holdingRECOMMENDED
Matches what vendor calling already runs; reverts instantly if evals dip. No — keep 0 for determinism
Keeps evals maximally reproducible; keeps every call identical.

H3 — A/B the voice itself?

Clara's voice ("Belle") is a professional voice clone that ElevenLabs says the expressive model doesn't fully preserve — we may be fighting the voice to get expressiveness out of it. Brand call, decided on recordings.

Run the A/B — 3–4 expressive-native voices vs Belle on real calls, Sean picks from recordingsRECOMMENDED Keep Belle regardless — voice continuity outweighs expressiveness

H4 — The prompt naturalness pass (personality on voice, varied lines, reactions allowed)?

Go — as its own PR series after the switches, judged-eval gatedRECOMMENDED
Pairs with the shared-voice-modules work in the One Clara architecture plan — the style layer gets written once, not eleven times. Fold into the One Clara migration instead
One less workstream; humanness waits on the architecture timeline. Hold

One Clara: cross-channel architecture — the structural companion; the shared voice-style module is where "more human" gets written once.
Aug 13 fix fleet (in flight): 7-second silence timeout, no repeated specials, name-only virtual tours, email persistence.
Carded: transfer double-acknowledgment fix · prompt-size ratchet · iron-law fence repair.