A CI suite ran paid ElevenLabs tests against the live Clara phone agents on every voice pull request, until the account was empty and customer calls failed at pickup.
Proposed — pending review · written 2026-11-03 · times in Central (UTC in brackets) · every number cites a PR, a file line or an ElevenLabs API read
On this page
TL;DR · Who added it and why · Cause chain · Timeline · Who did what · Cost · Which PRs burned it · Why no alarm · What stopped it · Guards (decision) · Not verifiedIn one line: a free "does Clara still answer right" check, wired to every voice PR in July, became a paid one on Nov 2 when ElevenLabs retired the free endpoint and we ported to the paid one without checking the price. Nothing watched the ElevenLabs balance, so it ran until the account was empty.
Added Jul 18 by #4103 (fede-propflow, a Claude session under Fede's account), "per Fede's directive: build the harness FIRST", with "nightly full sweep + change-scoped PR runs, soft (report-only)". It was meant to be cheap: "LLM tokens only — no TTS minutes" (voice-comprehensive.yml:4-8).
Made expensive Nov 2 07:10Z by #24185 (gera-propflow, a Claude session under Gera's account). ElevenLabs retired the free endpoint on Oct 31 (#24178); #24185 moved the suite to the paid agent-testing API. It maps every feature of the old API and never names a price: credit, cost, price, billing, usage, quota and spend appear nowhere in #24178, #24182, #24185 or their 34 comments and reviews.
voice-harness:* test runs (blue). Red: runs that ran at zero credits because the account was empty. Source: GET /v1/convai/test-invocations, every row from Oct 31 to Nov 3 20:01Z.| When (CT) | UTC | What happened | Source |
|---|---|---|---|
| Jul 18 | Jul 19 01:41 | Suite added with nightly + per-PR runs, "soft (report-only)" | #4103 |
| Jul 30 | 06:55 | Two app folders dropped from the PR trigger to save runner minutes (53 runs that day) | #4931 |
| Sep 29–30 | Nightly sweep moves to Mondays; PR trigger folded into the main CI workflow, same 7 paths | #13872, #14107 | |
| Oct 2 | 07:19 | Fede: "the live ElevenLabs tests are no longer necessary". Deletes 11 live test files; this suite is not a test file, so it stays | #15270 |
| Oct 31 4:37 PM | 21:37 | Free simulate endpoint answers 410 "endpoint_sunset" | #24178 |
| Oct 31 5:55 PM | 22:55 | First paid run, from #24185's own branch CI | invocations |
| Nov 1 9:17 AM | 15:17 | #24185 branch run: 424 paid runs in 32 min. Its branch ran the suite 5 times (~3.3M credits) | run 41320340373 |
| Nov 2 1:10 AM | 07:10 | #24185 merged. From here every voice PR push runs ~425 paid tests | #24185 |
| Nov 2 3:33 AM | 09:33 | Monday sweep: 569 runs, 1.28M credits | run 41421589799 |
| Nov 2 4 PM | 22:00 | Peak hour: 1,490 runs, 2.92M credits | invocations |
| Nov 2 8:38 PM | Nov 3 02:38 | Account empty. Runs keep going at zero credits until 07:50Z (4,286 runs) | invocations |
| Nov 3 12:23 AM | 06:23 | A robot call finds every Clara call dead | #24877 body |
| 5:12 AM | 11:12 | Health row reads "free plan, 10,000 of 10,000" | #24844 body |
| 8:37 AM | 14:37 | Pro plan period starts (derived: next reset is Dec 3 14:37:32Z) | /v1/user/subscription |
| 9:48 AM | 15:48 | Paid runs resume on PR pushes | invocations |
| 10:48 AM | 16:48 | First ElevenLabs credit alarm lands | #24877 |
| 11:19 AM – 1:57 PM | 17:19 / 18:54 / 19:41 | Empty three more times; usage top-ups lift the limit 600,000 → 4,236,360 | invocations, subscription |
| 2:01 PM | 20:01:36 | Last paid run | invocations |
| 2:04 PM | 20:04:30 | PR lane switched off | #24933 |
| PR | Account | Merged | Intent, in its own words |
|---|---|---|---|
| #4103 | fede-propflow | Jul 19 | "build the harness FIRST… CI: nightly full sweep + change-scoped PR runs, soft (report-only) until the gate set is proven green" |
| #4931 | fede-propflow | Jul 30 | "stop voice-comprehensive firing on app-side voice paths it cannot observe" (runner minutes) |
| #13872 | fede-propflow | Sep 30 | Nightly to weekly: "30 9 * * * → 30 9 * * 1 Mon" |
| #14107 | fede-propflow | Sep 30 | "Folds the pull_request arms… into ci.yml lanes… lane globs verbatim" |
| #15270 | fede-propflow | Oct 2 | "Fede ruled… the live ElevenLabs tests are no longer necessary" (test files only) |
| #24182 | gera-propflow | Oct 31 | "a sunset simulate endpoint fails the suite once, not per scenario" |
| #24185 | gera-propflow | Nov 2 | "Ports the voice-comprehensive / simulate job… to the replacement agent-testing endpoints." No price named |
| #24877 | gera-propflow | Nov 3 | "page #alerts when the ElevenLabs account runs low, runs out, drops to free" |
| #24933 | fede-propflow | Nov 3 | "switch the voice-comprehensive PR lane off" |
Correction: the cron comment and #24933's body credit the lane to #13737. #13737 only moved jobs to cheaper machines; the weekly move was #13872, the fold into the main CI workflow #14107.
Both accounts are used by Claude sessions; the PR bodies end "Generated with Claude Code".
| Slice | Runs | Credits | ElevenLabs price |
|---|---|---|---|
| Whole burn, Oct 31 22:55Z – Nov 3 20:01Z | 18,407 | 24.08M | $3,876 |
| runs that used credits | 12,570 | 24.08M | $3,068 |
| runs at zero credits (account empty) | 5,837 | 0 | $807 * |
| Before the account emptied (old plan) | 9,967 | 20.38M | $2,447 |
| On the new Pro plan, from 14:37Z | 4,154 | 3.70M | $819 |
| For scale: whole account, Oct 3–30 (calls and tests) | 3.24M |
src/lib/domain/spend/roster.ts:103-114). By 11:12Z Nov 3 it read "free plan". Most likely the burn used up the grant that was meant to last to mid-2027. Inference; confirm on the ElevenLabs billing page.| Live agent | Runs | Credits |
|---|---|---|
| Clara — Leasing | 7,753 | 13.30M |
| Clara — Triage | 4,611 | 5.66M |
| Clara — Maintenance | 2,227 | 2.44M |
| Clara — Resident Services | 883 | 0.62M |
| Clara — Leasing [WILLOWS] | 352 | 0.65M |
| Clara — Unknown Caller | 445 | 0.51M |
| Vendor Calling, Renewal, Coworker Line, Turnover Intake, Renewal Outbound, Handyman | 2,136 | 0.90M |
| Layer (scenario file) | Runs | Credits |
|---|---|---|
| regression (one scenario per past bug) | 11,030 | 15.51M |
| real (patterns from real calls) | 7,225 | 8.20M |
| adversarial + synthetic + architecture | 144 | 0.37M |
Largest name groups: leasing- 3,129 runs, real- 2,714, reg- 2,101, numbered bug ids (13380-, 12342-, …) 2,048, ws- 1,546, tool- 611. Tools were mocked (#24185: an unmocked tool raises an error), so no customer records were written; the harm was the shared credit pool.
| Account | PRs | Paid CI jobs | Runs | Credits | Price |
|---|---|---|---|---|---|
| gera-propflow sessions | 18 | 56 | ~15,616 | ~22.11M | ~$3,362 |
| fede-propflow sessions | 4 | 7 | ~2,221 | ~0.69M | ~$360 |
| Monday sweep (cron) | — | 1 | 569 | 1.28M | $154 |
Method: 1,998 pull-request CI runs Oct 31–Nov 3 were listed; 105 had a voice-comprehensive / simulate job, plus the Monday sweep; 64 of those 106 jobs ran paid tests (the rest failed fast on the retired endpoint, before #24185 merged, or were cancelled). Each test run was matched to the job running when it was created. 6,670 of 18,407 fell inside two overlapping jobs and were split evenly, so per-PR numbers are approximate. Two runs matched no job.
| PR | Account | Title | Paid/all jobs | Runs | Credits | Price |
|---|---|---|---|---|---|---|
| #24185 | gera-propflow | fix(voice-harness): port simulate job to ElevenLabs agent-… | 5/5 | 1,575 | 3.31M | $397 |
| #24516 | gera-propflow | fix(voice): Triage hears Maintenance-off only from the sta… | 4/4 | 1,308 | 2.63M | $316 |
| #24702 | gera-propflow | fix(coworker-line): a staff caller reporting a hazard page… | 4/4 | 1,743 | 0.86M | $286 |
| #24524 | gera-propflow | feat(bench): robot calls and robot texts refuse the Willow… | 4/5 | 1,063 | 2.09M | $251 |
| #24899 | gera-propflow | feat(voice): the staff phone line takes a repair request (… | 5/5 | 1,218 | 1.14M | $250 |
| #24886 | gera-propflow | fix(triage): a property manager walking a unit reaches Tur… | 5/5 | 1,103 | 1.43M | $248 |
| #24658 | gera-propflow | Coworker Line on at every company: retire the per-company … | 3/3 | 1,281 | 0.88M | $229 |
| #24352 | gera-propflow | fix(harness): bench scripts use the canonical test-propert… | 3/7 | 894 | 1.84M | $221 |
| #24636 | gera-propflow | feat(modes): a declared test is the DRAFT row; every actin… | 3/3 | 790 | 1.61M | $194 |
| #24732 | fede-propflow | refactor(scope): old company and building answers delegate… | 3/3 | 1,247 | 0.00M | $182 |
| — | cron | weekly Monday sweep (main) | 1/1 | 569 | 1.28M | $154 |
| #24619 | gera-propflow | fix(voice): a company-listed door that won't lock conferen… | 3/3 | 649 | 1.28M | $153 |
| #24485 | gera-propflow | refactor(site): normalizePhoneE164's importers import @pro… | 2/2 | 553 | 1.18M | $141 |
| #24807 | gera-propflow | fix(voice): an active leak after hours conferences in the … | 2/2 | 856 | 0.00M | $123 |
| #24617 | gera-propflow | feat(modes): "this is a test" from a teammate phone makes … | 2/2 | 464 | 0.93M | $112 |
| #24914 | fede-propflow | fix(voice): a closed office removes the leasing dial befor… | 2/2 | 484 | 0.51M | $104 |
| #24603 | gera-propflow | fix(tours): a building named as unit_number reads that bui… | 2/2 | 425 | 0.87M | $104 |
| #24544 | gera-propflow | fix(send-gate): a command-line run texts, calls or mails f… | 1/1 | 445 | 0.81M | $98 |
| #24436 | gera-propflow | fix(voice): no old-language word before the switch back to… | 3/4 | 392 | 0.80M | $96 |
| #24940 | gera-propflow | fix(voice-harness): refuse any shape or property that is n… | 2/2 | 420 | 0.29M | $77 |
| #24935 | gera-propflow | fix(voice): Clara never announces a connect nobody can tak… | 3/4 | 437 | 0.16M | $65 |
| #24946 | fede-propflow | fix(tours): another home's tour at the same time is a clas… | 1/2 | 406 | 0.17M | $63 |
| #24928 | fede-propflow | fix(voice): a name alone is an introduction, not an ask fo… | 1/1 | 84 | 0.01M | $12 |
Gera's sessions carried most of it because their open work (Coworker Line, robot-call guards, triage) touched voice prompts all day Nov 2–3. Nobody chose to run the suite; the lane fired on its own. #24732 and #24807 ran in the empty window, so their runs used zero credits.
v1/user/subscription only in a comment in the spend roster). No scheduled job on the Mac mini reads ElevenLabs. The spend page showed ElevenLabs as a fixed "$0, grant pays" line (roster.ts:93-114).elevenlabs-health.ts:46). Top-ups refill the limit, so a burn can stay above 10%.| UTC, Nov 3 | Action | Effect |
|---|---|---|
| 18:56–20:01 | Cancelled CI runs 41697767897, 41696636947, 41707250128, 41707804640 | Partly undone. The "CI follow-ups" rerun-if-runner-killed job and new pushes restarted them (41696636947 ran again 19:47–20:01) |
| ~19:52 → 20:04:57 | CI's ELEVENLABS_API_KEY secret replaced, then restored | New jobs could not reach ElevenLabs |
| 20:00–20:03 | Terminated 4 CI machines running the job (i-0cebf420b6e6802ad, i-0fe1ee27a7b8c9c89, i-0b8c098c1d1224216, i-00e70705abc898023) | Running jobs died. Last paid run 20:01:36 |
| 20:04:30 | #24933 merged: PR lane off (if: false && …, ci.yml:6651) | No new PR runs |
Recommendation: A, with the Monday sweep paused today. Any option also fixes the stale "LLM tokens only" header, and adds one line to the PR checklist: a port to a new paid API names its price.