PropFlow.ai PropFlow.ai Docs
Proposedmarkdown source

0048 — Iterate and promote the live voice agent via ElevenLabs branches, not agent clones

This ADR is both an architecture decision and a runnable playbook. A new engineer should be able to follow §Roadmap top-to-bottom and ship a change to the live voice agent without ever regressing Camellia traffic. Every step that mutates production is gated by a human approval and an eval-green check.

Context

Camellia's leasing voice agent is LIVE. Real prospects call +18445101007 and reach the shared Triage agent, which routes by transfer_to_agent to specialist siblings (Leasing agent_8601, Maintenance, Renewal, Resident Services). The live leasing path is therefore Triage → transfer_to_agent → agent_8601 — almost no prospect dials the specialist directly. The hard constraint is non-negotiable: never regress live Camellia traffic. A bad prompt edit, a dropped system tool, or a mis-pointed webhook is not a localhost bug — it is a missed lead or a falsely-confirmed tour on a real resident's call.

Earlier in this work-stream we tried to get a safe sandbox by cloning the agent (make a copy, edit the copy, point a test number at it, promote by hand-copying config back). Cloning failed in specific, reproducible ways (see §Pitfalls), all of which trace to one root cause: a clone is a fresh agent that does not inherit the original's tool bindings, override toggles, or phone wiring, so you rebuild those by hand and get them subtly wrong.

Meanwhile, ElevenLabs ships a native, git-style iteration model against the live REST API (https://api.elevenlabs.io/v1/convai). What is verified read-only (route existence + read shapes, confirmed 2026-06-04): branch list/read (GET /agents/{id}/branches → results[].current_live_percentage), branch create, branch edit without touching Main, and the merge route is path-style POST /agents/{id}/branches/{source_branch_id}/merge. What is NOT yet executed (documented, must be confirmed before any live write): the deployment/traffic-split write call and its percentage unit, and the merge call's body/behavior. (See §Flagged uncertainties — an earlier draft of this ADR mislabeled the merge shape and the traffic-read endpoint; corrected 2026-06-04.)

A branch inherits everything (system tools, webhook tool IDs, the enable_conversation_initiation_client_data_from_webhook override toggle, phone wiring) and Main is untouched until merge. That is strictly better than cloning. This ADR makes branches the default and demotes cloning to a fallback.

This decision also formalizes two patterns we learned the hard way and want to keep enforcing as we iterate:

  1. A testing ladder (sim → deterministic replay → real call) where each rung catches a different class of failure, so "the text sim passed" is never mistaken for "the voice agent is safe."
  2. A collect-live / enrich-post-call architecture (the maintenance pattern, generalized to leasing): do the minimum live, defer the rest to a post-call transcript-extraction step — which shrinks the live blast radius of every branch we promote.

The Triage-transfer routing question (must be resolved before Phase 3)

Because the live path is Triage → transfer_to_agent → agent_8601, a deployment split posted on agent_8601 only matters if transfer-routed calls actually get split-routed to the branch. Two open questions, both UNVERIFIED and explicitly gated in Phase 0:

We do not assert either behavior. Phase 0 includes a hands-on probe (place a Triage-transferred test call routed to the branch, then confirm the resulting agent_8601 conversation carries the branch's branch_id). If transfers do not honor the split/branch, the canary plan changes (e.g. split must be posted on the Triage agent, or the test must dial the specialist directly with that caveat documented). No Phase 3 ramp may start until this is resolved.

✅ RESOLVED (2026-06-05, from the ElevenLabs docs — supersedes the two UNVERIFIED questions above).

  1. transfer_to_agent runs the CHILD agent's own live config ("all other configurations are set by the child agent, including Prompt, LLM, tools…" — Personalization), and branch traffic is split deterministically by conversation_id (Versioning). The transferred call keeps its conversation_id, so a deployment split posted on agent_8601 IS resolved for transfer-routed calls — a Leasing canary is exercised by real Triage→Leasing traffic. (This is inferred by composing the two documented facts; still worth a one-call Phase-0 probe to confirm hands-on, but the canary architecture is sound, not blind.) The transfer carries dynamic variables, not the branch — only the child's own deployment decides which branch runs.
  2. The personalization-webhook branch_id does NOT work — at all, not just post-transfer. branch_id is not a field EL honors in conversation_initiation_client_data; it is silently ignored. The ELEVENLABS_STAGING_BRANCH_PHONES / _NUMBERS engineer-phone routing is therefore a no-op and must be dropped.

Net effect on this ADR: the traffic-split canary on agent_8601 (toolkit steps 5-6) is the correct, supported path and needs no Triage-side workaround. Drop the engineer-phone branch-routing wherever it appears below (toolkit step 4 / testing-ladder rung 3 "route your phone to the branch", line ~23/54): substitute dial the specialist's DID directly (+18442853526 → Leasing, as of 2026-06-05) or a small traffic-split %. See the companion playbook §4 for the full corrected mechanics.

Decision

We develop and promote voice-agent changes using ElevenLabs native branches + experiments + versioning + environment variables. Cloning an agent is deprecated to a fallback, used only when an agent has versioning disabled.

A process or code change violates this ADR if it does any of the following:

❌ Violation ✅ Required instead
Edits agent_8601 Main config directly to "try something" Create a branch off Main; edit ?branch_id=; Main stays live
Clones the agent to get a sandbox Branch off Main (clone only if versioning is disabled on that agent)
Points the live Camellia number at a candidate config for everyone Route only an engineer's dedicated-test caller_id (ELEVENLABS_STAGING_BRANCH_PHONES) or a small traffic % via a deployment
Promotes on a green text sim alone Pass the eval gate and a real voice/latency call before any live traffic
Promotes without the eval suite green run-tests green (intent-capture, never-confirm-on-error, no_prospect regression) is the promotion gate
Hard-codes a prod tool URL/secret into branch config so testing writes to prod Use environment variables; run tests/calls under environment=staging
Runs a "staging" call before the staging env-var gate (Phase 0) is green Prove every side-effecting tool has a non-prod staging value first — else "staging" silently writes to prod
Ramps traffic past a stage without a human "go" and a defined sample floor Per-PR / per-stage human approval and the sample floor in §Canary exit criteria
Lets a regression sit while debugging mid-incident Throw the kill switch when any hard rollback trigger fires (§Rollback triggers), then debug on the branch
Runs a second canary while another branch is mid-ramp One active canary at a time (§Concurrency); read /agents/{id}/branches (results[].current_live_percentage; /deployments is 405) before posting any split
Bypasses a failing pre-push hook with --no-verify Surface the failing test; fix-first / disclose / abandon — never silently bypass

(a) The branch-based isolation pattern (the core loop)

One agent_id, no cloning, no analytics fragmentation. Three primitives:

Primitive ID prefix What it is Mutability
Version agtvrsn_* Immutable config snapshot, auto-created on every save Immutable forever (rollback target)
Branch agtbrch_* Named isolated line of versions; every agent has a permanent Main Mutable pointer
Deployment — Percentage traffic split across branches (must total 100% across all branches) Reassignable instantly

The loop (each step is an explicit REST call). Every deployment write uses the full two-element array summing to 100 — never a single-item array (a lone [{branch, 5}] either errors or implicitly zeroes Main and blackholes 95% of callers):

Step Call Effect on Main / live traffic
1. Branch off Main POST /agents/{id}/branches {parent_version_id, name, description} None — new branch at 0% traffic, isolated
2. Edit the branch PATCH /agents/{id}?branch_id={b} (conversation_config / workflow / platform_settings) None — proven: branch first_message edit left Main unchanged
3. Read back GET /agents/{id}?branch_id={b} (include_draft=true for uncommitted) None
4. Route engineer's phone personalization webhook returns branch_id via ELEVENLABS_STAGING_BRANCH_PHONES None — only the listed (dedicated-test) caller_id hits the branch; everyone else → Main
5. Canary traffic POST /agents/{id}/deployments [{branch_id: <main>, percentage: 95}, {branch_id: <b>, percentage: 5}] — always include the Main entry; the two must sum to 100. ⚠️ Write endpoint UNVERIFIED — GET …/deployments returns 405, so the POST write shape/path is not confirmed; pin it on a non-prod agent in Phase 0 before any live split. Small % of callers routed to branch; deterministic per conversation_id (same caller, same branch)
6. Roll back POST /agents/{id}/deployments [{branch_id: <main>, percentage: 100}] (same UNVERIFIED-write caveat as row 5) Re-deploy Main 100% (propagation latency to be measured — Phase 0); or serve any prior version_id
7. Promote (merge) POST /agents/{id}/branches/{source_branch_id}/merge {target_branch_id=main, archive_source_branch} (path-style — route existence verified read-only 2026-06-04; body/behavior not yet executed) Mints new Main version, moves traffic to Main, optionally archives source
— Read traffic split GET /agents/{id}/branches → results[].current_live_percentage (verified 2026-06-04). Use this to confirm "Main at 100%" — not GET /deployments, which returns 405. Read-only

Percentage unit (MUST be pinned in Phase 0 before any deployment write). The roadmap writes whole-number percentages (5, 10, 25, 50, 90, 95, 100). It is UNVERIFIED whether the field is a whole number (50 = 50%) or a fraction (0.5 = 50%). If it is a fraction, percentage: 50 means 5000% and the API either rejects it or — worse — clamps/normalizes and mis-splits live traffic. Phase 0 pins this with a 0%-impact probe (post a split to the branch at 0 and Main at the max, read it back, confirm the echoed unit). Until pinned, no Phase 3+ deployment may run. All percentages in this ADR assume the whole-number unit; if the probe shows fractions, every value below is divided by 100 before use.

Isolation checklist (must all hold before step 5):

(b) The testing ladder (each rung answers a different question)

Do not skip rungs. A green sim is necessary but not sufficient — it cannot see voice re-fires or personalization.

Rung Mechanism Question it answers What it MISSES
1. Text simulation POST /agents/{id}/simulate-conversation (tool-mocked, read-only, fast) Tool count, intent firing, prompt behavior, eval criteria, false confirmations Voice re-fires (e.g. save_prospect ×3), personalization
2. Deterministic replay Webhook replay, per-tool Rigorous per-tool latency Turn-taking, ASR/TTS, barge-in
3. Real test call Route your dedicated-test phone to the branch (step 4 above), then pull the EL conversation (GET /conversations/{conversation_id}) Personalization + voice turn-taking (only rung that does); real tool sequence, per-tool latency, results, termination_reason — (this is ground truth)

Pulling a conversation (used in rung 3 and Phases 2–4). "Pull the EL conversation" means GET /v1/convai/conversations/{conversation_id} (list via GET /v1/convai/conversations?agent_id={id}&branch_id={b} to find the id). Read the per-turn tool calls, results, and termination_reason from the returned transcript. The per-tool latency field name (tool_latency_secs) is asserted from prior reads, not formally verified in the OpenAPI — confirm the exact field on the first real pull and fix references if it differs.

Rules baked into the ladder:

(c) The eval framework = the promotion gate

No branch reaches live traffic (step 5) until the saved eval suite is green. Built on the EL native testing stack: simulate-conversation + Tests API (type: llm / tool [verify_absence] / simulation with tool_mock_config) + run-tests with agent_config_override (test a modified prompt without mutating the agent).

Eval Type Pins Pass condition
Intent capture — clear intent simulation / tool Clear leasing intent ⇒ save_prospect fires save_prospect called once on clear intent
Intent capture — wrong number tool [verify_absence] "wrong number" ⇒ no capture save_prospect absent
Never-confirm-on-error llm / simulation On a tool error (e.g. schedule_tour → no_prospect) the agent must NOT confirm Transcript contains no tour confirmation when the tool errored
no_prospect regression simulation + tool schedule_tour self-resolves/creates its prospect schedule_tour does not return no_prospect; no false escalation
Triage transfer reaches Leasing simulation A leasing intent into Triage ⇒ transfer_to_agent to agent_8601 Transfer fires; the candidate branch is exercised on the real live path, not the specialist in isolation

Gate rules:

(d) Collect-live / enrich-post-call architecture

Shrink the live blast radius so each promoted branch can do less harm. Rule for what stays live:

A step stays live only if it needs real-time confirmation (a booking) OR comes from a volatile external source (tour slots from the property's Outlook calendar). Everything else is deferred to a post-call transcript-extraction step.

Domain Live (minimum) Post-call (enrich/discard)
Maintenance create_work_order collect-only maybeCreateWorkOrderFromTranscript
Leasing ONE optimistic save_prospect on clear intent (phone + property is enough → lead captured + a prospect exists for schedule_tour); check_availability; schedule_tour. Units are init-injected (no get_available_units). No live update_prospect. Prospect enriched or discarded from the transcript after the call

Correctness invariants this encodes (each from a real failure):

Pitfalls — EL config gotchas, and why branches avoid most of them

Pitfall What goes wrong Why a branch avoids it
both_tools_and_tool_ids_provided A request sends both prompt.tools (deprecated) and prompt.tool_ids → rejected. A cloned agent that holds both system tools and webhook tool_ids trips this. A branch inherits the original's exact ID-based bindings; you don't re-author them, so you can't reintroduce the legacy field.
Lost system tools A clone drops the system tools (end_call, transfer_to_agent, …) that live in built_in_tools. A branch carries built_in_tools forward unchanged.
1008 crash: "Missing required dynamic variables in tools" A bare clone crashes because the enable_conversation_initiation_client_data_from_webhook override toggle wasn't copied. A branch is expected to inherit the override toggle and the whole platform_settings — confirm by reading the branch config (Phase 0 toggle check) before relying on it.
Repointing the test phone number A clone is a new agent_id, so you must re-wire a DID to it. You route a caller_id to the branch via the webhook on the existing live number — no re-wiring.
Destructive platform_settings PATCH A wholesale platform_settings PATCH can clobber per-agent guardrails. Branch edits are isolated; and call_limits / privacy / auth are shared per-agent (cannot drift per branch) — guardrails can't silently diverge.
Override fields sent as ""/null Sending an empty override clobbers the agent default instead of falling through. Unchanged on a branch — but the rule still holds: omit fields you don't want to override; never send ""/null.
Env var missing requested environment Silently falls back to production — you think you're hitting dev, you're hitting prod (real SMS/booking). Branch + environment=staging is explicit; still: always define a staging value (enforced by the Phase 0 staging-env gate), and every var must have a production value (the only hard failure).
Versioning is irreversible Once enabled on an agent, versioning cannot be disabled. Not a problem in practice — it's the prerequisite for this whole pattern. Cloning is the fallback only if an agent lacks versioning.

Consequences

Positive

Negative / costs

Flagged uncertainties (confirm against live OpenAPI / hands-on before relying on them)

Rollback triggers (hard rules — not judgment calls)

The kill switch (re-deploy Main 100%) is mandatory — an engineer must throw it without waiting for sign-off — the moment any of these fire during Phase 2–4:

After throwing the switch: keep the branch (do not archive), reproduce on the branch, add the failing case as a committed eval, fix, re-green the gate, and re-enter at Phase 1. The alarm wired in Phase 0 evaluates these triggers automatically off the post-call webhook; manual monitoring is a backstop, not the primary trigger.

Concurrency (one active canary at a time)

The deployment array must sum to 100 across all branches, so two simultaneous canaries on agent_8601 interfere. Rule: only one branch may be in Phase 3–4 at a time. Before posting any split, GET /agents/{id}/branches (read results[].current_live_percentage; GET /deployments is 405) and assert Main is at 100% (no other branch live). If a second change is ready, it waits, or it is folded into the active branch and re-greened. This is a soft lock by convention plus the mandatory pre-split read in the isolation checklist.

Roadmap — the staged promotion pipeline

Small phases. Each stage has entry criteria, exit criteria, a kill switch, and observability, plus a human "go" at every boundary (per-PR / per-stage approval — never auto-ramp). Wire one variable per change (isolate the experiment). Live-traffic mutation steps (Phases 2–5) are Fede-executed until Gera's access is provisioned (see Phase 0).

Phase 0 — One-time setup + verification gates (per agent)

Setup:

Hard verification gates (each must pass before the phase it unblocks):

Phase 1 — Branch + eval-green (zero live traffic)

Phase 2 — Route the engineer's phone (live number, one caller)

Phase 2.5 — Controlled prod-booking dry run (isolate the highest-risk live path)

Phase 3 — Canary 5% (real traffic, small)

Phase 4 — Ramp 10 → 25 → 50 → 100%

Phase 5 — Merge / promote

Alternatives considered

  1. Clone the agent (the deprecated approach). Make a copy, edit the copy, point a test DID at it, hand-copy config back to promote. Rejected. It hit both_tools_and_tool_ids_provided, lost system tools, required copying the enable_conversation_initiation_client_data_from_webhook toggle (a bare clone 1008-crashed with "Missing required dynamic variables in tools"), and required repointing the test phone number. A branch inherits all of that and never touches Main. Kept only as a fallback for an agent with versioning disabled.

  2. Edit Main directly behind a feature flag / off-hours. Rejected. No isolation — a mistake is live immediately on a real resident's call; rollback means another live edit; and analytics can't cleanly attribute before/after. Branches give isolation, config-only rollback, and clean attribution.

  3. Stand up a separate staging agent on a separate number. Rejected as the primary path. It re-introduces clone-style drift (two configs to keep in sync, tools/toggles to re-wire) and never tests the actual live number's personalization path. A dedicated test DID pinned to a branch via PATCH /phone-numbers/{id} (branch_id + environment) — i.e. whole-line routing via ELEVENLABS_STAGING_BRANCH_NUMBERS — is available as a complement for whole-line test traffic, but it carries the same eval gate (no line may be pointed at a branch without Phase 1 green) and is otherwise unused in the default flow. The engineer-phone-on-the-live-number route is the higher-fidelity default; this alternative exists so the dangling ELEVENLABS_STAGING_BRANCH_NUMBERS capability has a defined, gated role rather than inviting an ungated whole-line repoint.

  4. Promote on text-eval green alone (skip the real call). Rejected. Text tests are voice-blind: they miss voice re-fires (save_prospect ×3), ASR/TTS, turn-taking, barge-in, and personalization. The ladder requires a real call (rung 3) before any live traffic.

  5. Random per-call traffic splitting. Rejected — EL routing is deterministic per conversation_id by design, which gives consistent per-caller UX. We accept the trade-off (a single test caller can't randomly sample branches; canary cohorts skew toward repeat callers) and mitigate by restricting A/B comparisons to first-time conversation_ids and relying on real canary volume over the sample floor.