Video Clara coworker

Staff talk to Clara on video, either from our website or by inviting her into their own Zoom, Meet or Teams meeting. In a meeting she mostly listens. After the meeting, what the team said about each tenant goes to the renewals engine as notes for staff to approve. The near goal: Clara sits in on a client's renewals meeting next week.

Recommended: video is a fourth channel into Clara's one engine. It isn't free: that engine waits for the whole answer before speaking and loads a huge prompt for staff. Five changes fix that; dev measurements (Sonnet 4.6, two runs each) put her first word at 0.7–2.9 s.

Proposed — pending review. Fede, 2026-10-07, after the 3:55 PM client meeting: "Clara just couldn't talk at all… I don't think the current design is going to work for a meeting with a lot of interruptions."

What happened. One-on-one Zooms worked. The group meeting did not: Tavus heard the whole room as one speaker and never found a gap.

Why. Three things stack up. The live persona is set to high interruptibility, so any sound stops her. The wake phrase the design called for was never set on the persona, so she tried to answer everything. And Tavus gives us no per-speaker audio in a meeting, so our engine cannot tell who is talking or whether it was addressed; it only sees one merged "user" turn. (Sources: persona p3f51613c91e read 2026-10-07; Tavus conversational-flow and meetings docs.)

What Tavus offers. Interruptibility low / medium / high ("low: keeps talking through minor interruptions"), turn-taking patience, the sparrow-2 turn model ("fewer interruptions in noisy settings"), and wake / sleep phrases. There is no "never interrupt" or "finish the sentence" mode, and nothing documented about several speakers.

Applied 2026-10-07 18:35 MT on Fede's go ("go with low", "go with all the best for group settings"): live persona p3f51613c91e now has interruptibility low (was high) and turn-taking patience high (was medium). Wake phrase not set yet: the persona is shared with the website one-on-one call, so a wake phrase there would silence Clara until "Hey Clara" on every call. Next retest: a 3-person Zoom with crosstalk.

Recommendation. Do A today as the go/no-go, with the retest recorded here. If she still cannot finish a sentence in a 3-person crosstalk test, go to B for meetings and keep Tavus for the website call. Build C either way: it is the shape that works in a room full of people.

Other bugs from today's practice calls (2:08 and 2:16 PM one-on-one Zooms, from the transcripts):

• Refused to add up delinquent balances until pushed ("Are you sure you can't just add them up?").
• Described the person on camera wrongly ("a woman with brown hair"), then read a name tile as "Cera Garza". Vision is on in the persona although the design says off for client meetings.
• Reported a black screen with a Zoom notice until the testers rejoined; our code sets nothing about vision, so this came from the Zoom side.
• Could not say whether a tour was self-guided, where work orders came from, or collections status.
• No way to remove her from the meeting from the website. Fixed and live (#17579).

Possible, but tight. A staff-only meeting where Clara listens and answers basic questions needs 6 of the 9 engine PRs, most of a seventh, and the meeting join itself, all proven at Fairhaven first.

Fallback if it slips: Clara joins and only listens. She answers "I'll follow up after the meeting," and the transcript still feeds renewal notes. That needs just the binding, the meeting join and the transcript callback. Proposal.

v1 meetings are the client's own staff only. Tavus docs don't say what attendee or invite details Clara sees, so she can't detect outsiders yet.

Program plan: the Western Slope pilot (notes → brain → renewals) lives on the company brain page.

Proposed — pending review. Fede asked 2026-10-03: "can i test on zoom?" → "scope it".

How she joins: Tavus can send Clara into a Zoom, Google Meet or Teams link. We pass the link as meeting_url when we create the call, and her face and voice join as a normal attendee. No Zoom app review is needed for that path. (Tavus API, Tavus FAQ)

Already built for the website call and reused: creating the Tavus call, the company binding, Clara's answer line, the company snapshot, the end-of-call callback, Sonnet 5.5, and the reply timer.

Five unknowns the Tavus docs don't answer (all say "not stated"):

1. Does our own answer line (Sonnet 5.5, our data) still drive Clara inside a Zoom? Blocker if no: Fede ruled no Tavus-hosted brain with customer data.
2. Does she get each speaker's name?
3. Is the call still capped at 15 minutes on our plan?
4. Do the wake and sleep phrases work in a meeting?
5. Do waiting rooms or passcodes stop her? Zoom has required approved tokens for bots in other companies' meetings since March 2, 2026 (Zoom); whether that touches Tavus's attendee is untested.

If unknown 1 fails: a meeting-bot service carries the audio and our own engine answers. The candidates are Recall.ai (about $0.50 an hour; pricing) and the open-source Attendee (GitHub). That path is bigger, and Clara probably has no face in the meeting.

Video is a fourth channel into Clara's one engine, the same loop that answers texts and emails. Each spoken turn enters the engine directly, the way staff clarifying questions already do. The speaker is the signed-in staff person from the call binding, not a phone number. Ask Clara's 23 portfolio reads become one capability of that engine, STAFF_PORTFOLIO. In v1 a video turn can only read: a new gate allows read tools only. AppFolio data goes into a snapshot built when the call starts, under the per-account request budget, never as a live AppFolio call on each question.

The honest cost. Fede's direction is right, but it isn't "nothing to rebuild." Today the engine doesn't stream, a staff question loads about 57k tokens of leasing prompt, and there's a long setup before the model is called. Five changes make it fast enough to talk to:

1. Stream the answer as it's written.
2. Staff questions stop loading the prospect leasing prompt.
3. Build a snapshot and warm the cache when the call starts.
4. Use the voice model tier.
5. Cap each turn at 2 lookups with a deadline of about 6 s.

A sixth change, skipping the resident and prospect setup steps, is needed only if a bench trace shows that setup is slow.

Dev measurements: Fairhaven test company, N=2 per cell, warm cache, with the changes above in place. Lookups on a one-property dev company took under 60 ms, so a Western Slope-sized company will be slower. Without the changes, the estimate is 4–6 s to the first word on a question that needs a lookup (inference).

Proposed — pending review. Fede asked for a version that runs entirely on Tavus (their recommended model, tavus-gemma-4, plus their knowledge base), to see if our own brain is the ceiling. Same face, same voice (Belle). The Tavus version reads a made-up company; ours reads Western Slope. Measured live in Tavus's own room: time from question to her first word (typed question, so microphone pickup isn't included for either).

What it shows: the speed gap is the brain, not Tavus. Tavus's own model starts talking in about 1 second. But its knowledge base looks up passages; it can't count or list, so it guesses, and it guessed wrong on 5 of 6 facts. It also has none of our live lookups, note-taking or privacy checks.

Found while testing: Tavus's fastest lookup mode ("speed") makes the hosted brain go silent; "balanced" works. That goes on the Tavus question list.

Built and off everywhere: company setting clara.videoBrain (default ours). Tavus-hosted test persona p2c6a736c5c4, made-up data only.

Proposed — pending review. Fede asked Oct 6: "an audit ... with the tools it has access to and what's missing. I want the entire PropFlow platform included ... Long term I'd like it to be able to access every single module's data and all entities." Read from the code on main, Oct 6.

  Call start ──► summary of "what's happening now" (built once, free on every turn)
  Each turn ───► 22 lookup tools (read-only) + live AppFolio reports
                 each lookup adds ~2–6 s to that answer

What it shows: every existing Ask Clara lookup is already on video, so new reach means new lookups, not switching old ones on. Biggest blind spots: the knowledge base (pets, parking, policies), households, renewal offer details, payments received, the calendar and waitlist, and inspections. She can already pull live AppFolio reports (delinquency with late counts, move-out reasons, ledgers), but slowly and only for AppFolio companies.

Constraint: her call-start summary takes 19–31 s to build against a 30 s limit, so most new data has to be lookups, not more summary.

Decided Oct 6 (Fede): narrower than the list below. Build: move-out reason, who's behind on rent, lease terms. Then all the basics for leasing (prospects, tours, application status), maintenance (how many open, any ticket's details) and renewals: counts in her call-start summary, details through the lookups we already have. Not now: turnovers, building knowledge, the rest of the platform.

Where to expand first

Long-term shape (toward "every module"): a thin summary of what's happening now, plus one read-only lookup per module, each checked as read-only before it's allowed on video. Kept out on purpose: login credentials, phone-line setup, staff accounts and sync internals.

Decided Oct 6 (Fede, after a live meeting test): "let's make this the permanent version and remove the feature flag and the old version... this becomes a long-term architecture." Our own brain is the only video brain. The hybrid and Tavus-hosted versions, and the company switch between them, are being deleted. In that meeting Clara answered in a median 1.8 s on our server across 27 turns.

Being fixed now, from the same meeting: (1) she said "Sorry, I couldn't get that" three times when Fede was talking to himself; she should stay quiet. (2) She listed October offers when asked about November; she should answer only what was asked. (3) Basic renewal and tenant questions: who gave notice and why, the renewal increase, how many renewals are open, a tenant's balance and late-payment history.

Proposed — pending review. Same 10 questions, same test company (a copy of Western Slope's buildings, no real customer on the call), same face and voice. "Hybrid" = Tavus's model talks and asks our server for every number (option B above). Answers scored against the real data, read with the row cap off.

What it shows: the hybrid starts talking twice as fast and is about as accurate. Why the misses happened: the hybrid's December miss was a cut-off list (verified: 27 KB of leases cut to 8 KB). Its renewals miss was a lookup that came back empty, likely a status-name mismatch (every renewal is marked "ready"). Our misses go through a different path; cause not checked yet.

Hybrid lookups: 8 calls in 10 questions, all succeeded. Our server took 0.3–2.0 s each (median 0.8 s), over the 0.3 s target; the slow ones are the big lists. Hybrid isn't yet told to stay quiet on chatter; that's its own small change. It still runs the instructions it was created with on Oct 5, not today's video instructions.

Ours, after: tonight's fixes were silence, screen share and counts, not speed. It is no faster, and the slowest answers were slower (8.3 s vs 5.9 s), on a different data set with more questions. "Before" was on the real Western Slope data, "after" on the test copy: compare speed, not facts.

Proposed — pending review. Five minutes, Western Slope Clara in Fede's team Zoom. Sources: the Tavus transcript, our server logs for the call, and a read of the live renewals data.

Speed on this call: first word after she was asked: 1.6–3.6 s on most turns, 4.2–5.6 s on 4 (cold start, a fresh server, one data lookup). No errors, no timeouts.

Proposed — pending review. Fede's framing, Oct 5: the face mostly mirrors what is said, so video Clara is a voice agent. We already know how to run voice agents on ElevenLabs. Treat Tavus as one more provider that adds a face. The only real difference is the audience: phone Clara talks to people outside (tenants, prospects, vendors), video Clara talks to people inside (coworkers).

The flaw today: phone and video are two different machines. Phone runs on ElevenLabs with rules written for calls. Video runs on Tavus with our email and text rules plus one line asking her to be short. That is why she read seven renewals aloud and spoke her private notes on the Oct 5 Zoom test.

Three long-term paths

Recommendation: B, in three steps.

1. Now (in flight, no customer effect): video takes the shared call rules, stops speaking private notes and the "one sec" filler, and gets a test bench that checks answers without a video call. All of this carries over to B unchanged.
2. Two cheap probes (~5 Tavus minutes, needs Fede's go): (1) Tavus Echo Mode inside a Zoom meeting, fed audio from an ElevenLabs test agent; (2) ask Tavus whether a custom model's reply is spoken sentence by sentence, and about the choppy audio on conversation c67ab293976cc446. If Echo can't join Zoom, the fallback is a voice-only meeting bot.
3. Then: the coworker specialist on ElevenLabs, same settings and voice as phone, with Western Slope's company snapshot loaded at call start. The face becomes a per-meeting choice.

Unverified: whether Echo Mode works with a meeting link (docs show it in Tavus's own rooms); whether ElevenLabs agents can join meetings natively (none found; the documented route is a meeting bot). Phone's model runs at ElevenLabs today, so B also means company-level staff data reaching ElevenLabs.

Proposed — pending review. Measured on two test calls the night of Oct 4–5; nothing below is being built until Fede picks.

Short answer: the model is the biggest single piece (1.2 s, measured). The likely surprise: on 12 of 12 turns, Clara's voice started 0.57–0.73 s after our whole reply had arrived, never during it. That points to Tavus waiting for the full reply (~0.7 s per answer), but it is not proven: every reply was about the same length (~100 tokens), so a slow voice engine would look the same. The 3-second pause test in move 1 settles it.

What we already did (Oct 4–5)

• A dedicated fast route for video answers.
• Thinking turned off.
• Send at the first phrase, not the full sentence (no gain until move 1 below).
• Company data loaded and cached before anyone speaks.
• The conversation is reused turn to turn. Proven: the cached part grows each turn, and the new part stays at ~130–270 tokens.
• Meeting-first turn-taking: wake word "Clara", interruption sensitivity high.

Tavus test minutes used: 6.04.

Top 3 moves

Instant opener (~1.9 s off the first sound) only works once move 1 is fixed; today Tavus would hold the opener until the whole answer finished.

Ruled out

• Haiku or any smaller model. Fede, Oct 5: no.
• Moving company data onto Tavus's own AI. Customer data can't go there.
• Shrinking the prompt. Measured: the first word takes ~1.0 s with a warm cache whether the prompt is 5k or 33k tokens.

Decisions for Fede

Risk on (c): low patience may cut people off mid-sentence in meetings. The wake word "Clara" softens it, because she only answers when addressed.

Short answer: data isn't the delay. On the test link all of Western Slope's data is already loaded into her instructions when the call starts, so a normal question reads nothing (0 ms), and her words stream to Tavus as they're written. That's the same "everything in the prompt" approach the phone agent uses, with the same model. The extra time is on the video side: Tavus deciding you've stopped talking, an extra internet hop to our server, and rendering the face.

What Fede saw on the 8:45 and 8:51 calls (2026-10-02), and what changed the same morning:

Where data should live, as companies grow:

This is the scalable version of "load everything into the prompt": load everything about this caller's buildings, cached, and keep lookups for the long tail. A 10k-unit company still gives each manager a small summary because it's scoped to their buildings.

Proposed — pending review. In production, Clara writes the whole answer before Tavus starts speaking. Streaming it word by word is a chain of three changes, not a small one: (1) a streaming version of the shared model call that every Clara channel uses, (2) a decision on three safety re-checks that today re-write a finished answer (they can't re-write words already spoken), (3) a rule for the two steps that clean an answer before it goes out (removing internal instructions, and the personal-information check). Options:

1. A. Speak the finished answer sentence by sentence after the cleaning steps. Small change, saves a few hundred milliseconds.
2. B. The full three-change chain. Biggest gain, roughly matches the test link's speed.
3. C. Leave as is.

Long-term setup (Proposed — pending review)

One Clara brain, on our side. Tavus (and later the phone vendor) only carries the face and voice. Speed comes from streaming, a warm prompt cache, running in the cloud next to Tavus, and trimming setup.

Ruling (Fede, 2026-10-02): no smaller (Haiku) model, and no Tavus-hosted brain with customer data. The Tavus-hosted model is used only as an A/B measurement, against a made-up company ("Pinecrest"), so no customer data goes to Tavus.

Staying on the $59 plan: Clara rejoins every ~15 minutes. Starter caps each Tavus call at 15 minutes, and a renewals meeting runs longer. Whether Starter can join meetings at all is unverified: the pricing page and the meetings doc don't say which plans can.

• Our own meeting session is the parent. The binding and Clara's memory hang on it, not on a Tavus call.
• Each Tavus call is a child of that session. Its company is copied from the session when we create it.
• At about 14:30 our server ends the child at a quiet moment (nobody addressing Clara) and starts the next one.
• Running notes carry forward on our side. Tavus memories are never used.
• Gap risk: a few seconds of absence plus the join animation, and anything said in the gap goes unheard. We measure it from the end time of the last child to the first transcript line of the next, every rejoin, and report the median and worst case.

This is the most dangerous part. One rule: only a signed-in person starting Clara from our app names the company. Every other signal can only make her say less.

What Tavus sees: we upload nothing to Tavus. It hears what Clara says on the call. Recording and memory are off, and each conversation is deleted after the call.

v1 (next week): start from our app only; the client's own staff only, because Tavus docs don't say what attendee or invite details Clara sees, so she can't detect outsiders yet; read-only; the start page warns that Clara will share this company's data with everyone in the meeting. Later: joining from a calendar invite (with a verified organizer and the attendee veto), automatic no-specifics mode, actions, and owner meetings.

Clara is the ears; the renewals team owns what happens to what she hears.

• After the meeting, the transcript comes to us. The renewals lane's extractor pulls out, per tenant, what the team decided: lease length, price, rent, exceptions, and who on the lease it concerns.
• Nothing applies until a person approves it, through the renewal approval step that already exists.
• She also learns how a team walks through renewals ("October has three, then each tenant"). That comes from real meetings, not from us guessing.

Proposal from the renewals lane (to be decided with Fede after Western Slope's renewals turn on Monday; date unverified here): no separate store. The notes become a per-lease, per-renewal-cycle override that the renewal policy reads, keyed the same way as the renewal record (lease plus cycle).

Source of real meetings: we have no Camellia tenant-by-tenant renewal-meeting transcripts. Western Slope holds a weekly recorded renewals meeting and offered to invite us (unverified here).

"Send this renewal, but make this exception" goes through the same staff tools and the same write gate as a staff text. Clara repeats the action back and waits for a spoken yes before doing it.

Honest gap: staff texts today have no built-in "confirm before write" step. Desk writes run as soon as the model calls them, and the prompt only asks Clara to read back the result (staff-write-gate.ts, staff-leasing-desk-prompt.ts). We build that confirm step once, in the shared staff capabilities, for text and video together. A video-only version would be a parallel path.

Video Clara is run like the phone agents: her setup lives in our code, a merge deploys it, and a nightly check catches hand edits. The setup Tavus holds is thin and has no client data; what Tavus hears on a call is covered under Keeping companies apart. Everything about which company is on the call lives on our server.

Two rules that come from how Tavus behaves:

• Setup changes go in their own PR, never mixed with endpoint code. Tavus test-calls our answer address when a setup changes, and our code can reach production up to 30 minutes after merge.
• Creating the first production setup is a third-party production write, so it needs Fede's go. After that, the merge lane updates it.

The Western Slope production snapshot used by the local proof never goes into git or CI. Tavus docs put the wake phrase on the setup, and create-conversation lists no per-call override, so meeting mode likely needs a second setup per environment (inference). Source: build-deploy findings, 2026-10-01, from origin/main e6be230d12.

The code is done. Clara can take a video call, answer from the same brain as text, email and phone, look things up only inside the caller's buildings, and close the call cleanly. Nothing is on at any company yet.

Streaming, reopened 2026-10-02. Most whole-reply checks are off for staff, but three re-checks that re-write a finished answer and the two cleaning steps still run, so true word-by-word streaming is a chain of three changes. The plan is in Why video feels slower than the phone.

Skipped for now: the full staff write list (no exposure: video can only read).

What's left, in order:

1. Fede: yes to create the production Tavus persona and set its three secrets. (Tavus Starter bought 2026-10-02.)
2. Turn video on for our own test company (Fairhaven / Willows) and run a 5-minute bench call: every answer traced to a real record, first-word time read from the logs, the end-of-call signal seen.
3. Fede turns it on for Western Slope's website.
4. Meeting join with the 15-minute rejoin, about a day after Starter.

Follow-ups (not blocking): a "Talk on video" link in the app, a company picker so admins can start a call, the outcome event for company-level calls, a race when the call ends mid-answer, and the eval proof for the call-start summary (#15488).

Open, for Fede:

Decided by Fede on 2026-10-01: Clara's phone voice (ElevenLabs), a Tavus stock face with no real person, Tavus account set up, answers only in v1, website first. Updated the same night: also join a client's renewals meeting next week. Video is another modality of the one Clara, not a wrapper around Ask Clara.

• Staff questions load the prospect leasing prompt. A staff-only turn pulls in about 57k tokens of leasing prompt through the "unknown sender means leasing" fallback (leasingBodyWouldLoad, clara-unified.ts:85–111). This is likely live on today's staff texting desk too (inference; one desk trace will confirm). It's chain step 1, a fix with no switch.
• Staff write list: a small consistency cleanup, no exposure.

• Cross-client leak in a meeting. This is the top risk. See Keeping companies apart.
• AI disclosure in client meetings. Attendees may not know she is AI, and EU rules on this apply from 2026-08-02. Flagged for Fede; not decided here.
• Call length. Free plan calls stop at 5 minutes (our proof calls ended at the cap), and Starter calls stop at 15. Rejoin covers this, with a gap.
• Which plans can join meetings: unverified. The pricing page and the meetings doc don't say. Check in the Tavus dashboard before buying.
• No recording on meeting joins. Tavus records only its own rooms. We rely on the transcript.
• Tavus analyzes faces. One proof call returned a description of the caller's appearance. We turn perception off for meetings.
• Speed. The first-word times are dev measurements on a tiny test company. On Western Slope-sized data, lookups will be slower and she may feel laggy.
• Tavus hears every answer. Zero-retention is Enterprise only. Recording and memory off; we hard-delete each call after storing the transcript.

Sources: Tavus create-conversation docs; recall.ai/pricing. Recommendation: Tavus with rejoin for next week; a meeting-bot service is the long-term option for meetings where Clara mostly listens.

Research passes 1, 2 and 2b (2026-10-01), covering part of the ~230 doc pages. More research is coming and will be added to this section.

Overage is $0.37 a minute ($0.32 on Growth), billed with a 30-second minimum and then in 6-second steps. Third-party comparisons put Anam at $0.11–0.16 a minute. Sources: tavus.io/pricing, docs.tavus.io (llm, meetings, conversational-flow, webhooks, create-conversation, tools), research file pass 2.

Not documented yet: the bot's display name, waiting rooms and passcodes, what ends a meeting call, how speakers are told apart, the attendee list, and API rate limits.

• Properties and units
• Tenants and leases
• Prospects and tours
• Renewals
• Work orders and turnovers
• Conversations, including emails Clara was part of
• AppFolio reports, through our adapter
• Basic leasing questions

Not yet: taking actions such as booking a tour or adding a note. See Actions on a call.

1. Local test with Fairhaven test data.
2. Built into the website behind a company setting that is off everywhere.
3. Fede tries it with Western Slope data.
4. A test meeting on our own Zoom, then Western Slope's renewals meeting, on Fede's go.
5. Turned on at Western Slope only, on Fede's go.

See the plan table under Tavus: everything we learned. With rejoin, a one-hour meeting is four Starter calls, about 60 of Starter's 100 monthly minutes. Buying a plan is Fede's call.

1. The engine chain: done; see Where the build stands.
2. Meeting listener: the meeting session, wake phrase, 15-minute rejoin, and transcript to renewals.
3. The "Talk to Clara" button, with the video call right on the page.
4. Deploy lane for the Tavus setup, built the ElevenLabs way.

Vision — Fede, 2026-10-02. Not being built; written down so today's design choices don't block it.

Each customer gets a company brain that Clara draws on.

• Base layer (parts exist today): knowledge base, policies, listings, fees, rules.
• New layer, coworker notes: what Clara hears in meetings she joins on video or Zoom, transcripts and notes the team shares with her, and team decisions. It could be part of the knowledge base or a separate store next to it.

Clara checks this layer before acting. Example: renewals. Today the renewal flow follows basic rules. With the brain, Clara first checks whether a team meeting or decision changes what she should do.

It is an add-on, so not every customer will have it.

What it means for today's video work: meeting transcripts are worth keeping per customer rather than throwing away, and they never cross between customers.

Tracking issue: #15255. Sources for this page: origin/main dfeb59b5cc (inbound-router.ts, envelope.ts, conversation-manager.ts, capabilities/index.ts, dashboard/chat), ADR-0022, ADR-0054, architecture-source-of-truth (P5, P8, F7), portfolio-architecture-how (§02, §04, §05), agent-loop-decoupling-2026-09, voice-agent-architecture-decision, docs/runbooks/elevenlabs-voice-agents.md, docs/runbooks/renewal-orchestration.md, and the local proof log.