Fast-brain migration: day-one numbers

The one-paragraph version

Day one of making the fast brain launchable. The full script rewrite fixed the two bugs that mattered most — silently booking times the caller never chose, and claiming a booking succeeded when the system actually failed — both verified line-by-line in transcripts, not by automated scoring. It also introduced four smaller new problems (three of the four already fixed in same-day follow-ups — details in section 4). On the live phone line, 19 real calls through the actual transfer produced 0 unasked bookings — details and the honest limits of that number below. And the anatomy of the booking eagerness is now clear: the urge to act without asking is the model's own behavior (script fixes reduced it substantially with the tool untouched) — but the booking tool's definition requires a date and time, which guarantees that whenever the urge wins, the model must invent values. The permanent answer is structural: reshape the tool and add a server-side hard-stop, because script rules alone measurably top out short of 100% on this model.

1. The live phone line: real calls through the real transfer

This is the measurement that matters most, because the "books without asking" bug has only reproduced on real calls — 60 attempts to recreate it in the simulator, using the exact data from Fede's failing call, produced zero failures, while Fede hit it at least twice on the live line (the original discovery Wednesday night, and Friday's call — transcript on file). Why real calls differ is a working hypothesis, not proven: the leading explanation, supported by a controlled experiment where adding the "let me pull up your tour" lead-in raised the silent-booking rate from 0 in 5 to 3 in 20, is that the agent treats that just-spoken line as momentum to act. So we measured where the bug actually lives: the real line, with scripted robot calls from a dedicated test number, each one a genuine "I'd like to reschedule" with no time given.

Read honestly: 19 clean calls rules out a bug that fires most of the time, and puts a ceiling of about 1-in-7 on the true rate (95% confidence). It cannot distinguish "the ask-first fix closed this" from "it still fires somewhere under 15% of the time and 19 calls didn't catch one" — Fede personally hit it twice this week, and a separate simulated probe of the fixed script still reproduced the silent booking in 1 of 3 runs. That residual uncertainty is exactly why the server-side hard-stop below exists: it doesn't need the rate to be zero, it makes the outcome zero.

2. Old script vs new script, replaying real production calls

36 test scenarios were built directly from real recorded calls (mostly Camellia production traffic). 13 had to be excluded because the simulated caller wouldn't stay on script — a testing-tool limitation, excluded identically on both sides, disclosed not hidden. On the 23 that scored cleanly:

Flat on the surface — but the composition changed in the direction that matters. The one scenario reproducing the confirmed production bug (a real caller said only "morning tomorrow" and got silently booked for 9:00 AM) flipped from fail to pass: the new script stated a concrete time and got the caller's agreement before booking in that round's runs. (Later, larger sampling put this fix at 6 of 8 runs — see section 4; real, not yet bulletproof.) The two scenarios that flipped the other way are real but lesser: one self-contradiction on an ambiguous-property call, one hand-off done through the wrong mechanism with correct content.

3. The two flagship fixes, verified by hand

• Wrong-time confirmations: fixed. With the booking system rigged to return a different time than requested (10 runs), the automated grader initially scored the new script worse — but reading every transcript shows the grader was penalizing the new script for doing the right thing (restating the request to confirm it). In all non-looping runs, the last thing the caller hears is the true booked time. The old behavior — repeating the caller's requested time as fact — is gone.
• Honest failure handling: fixed, 0 of 5 → 5 of 5. When the booking system errors, the old script told callers "that slot just got taken" and implied things were fine. The new script says plainly the slot isn't available and offers alternatives — five out of five runs, verified in transcript.

4. New problems the rewrite introduced — and where they stand after two more rounds

Four new defects surfaced in the first test round. Two follow-up script versions (each a minimal, measured change) closed most of them the same day:

• Self-contradiction on ambiguous property — fixed (3 of 3 clean after adding "commit to a stated assumption after one unresolved ask").
• The stuck loop that never books — fixed (was 2 of 10; now 0 of 10 after promoting "never hold up a tour for missing details" to a standalone rule).
• The hallucinated gym — fixed, on the third attempt. A polite per-item rule failed twice (the model kept asserting a gym against a list with none — its "apartment complexes have gyms" instinct beat the list). What worked: forcing the answer to begin by quoting the property's literal amenities list, with the gym trap named explicitly. 3 of 3 clean, with the mechanism visibly firing in transcripts.
• Wrong transfer mechanism — fixed for its target (refund requests: 3 of 3), but the generalized rule still misses an adjacent case 2 of 3 times — the one remaining open prompt item.

And the ceiling, measured: the flagship silent-booking fix holds in 6 of 8 pooled runs (one apparent failure was a test-rig artifact; one was a genuine recurrence — "sometime tomorrow morning" silently became a 9:00 AM booking). Around 75% is where script-only fixes plateau on this model. That number is the whole argument for the next section.

5. Root cause of the booking eagerness — and the permanent fix

Three verified facts now define the fix:

• The booking tool guarantees invention once the model decides to act. Its definition marks the new date and time as required — the model literally cannot call it without values, so on every observed failure it supplied values the caller never said (a slot from the offered menu in the original incident; an invented time in one controlled run with the menu removed). To be precise about causality: the tool doesn't create the eagerness — script fixes reduced the failure rate with the tool untouched — it guarantees fabrication whenever the eagerness wins.
• Prompt rules alone don't fully bind on this model. The "always ask first" patch was verified word-for-word in place on the live agents when Fede's Friday call failed anyway, and the rewritten script still shows 2 misses in 8 runs on the hardest scenario. Industry research found no one who claims prompt emphasis solves this class.
• The only guaranteed layer is our own server. The plan, per industry practice and Fede's direction: reshape the tool (the required fields become "the caller's own words for the time," which the model must quote), and add a server-side check that refuses any booking not backed by caller confirmation — telling the agent to go ask. The check adds no delay when the agent behaves; it costs a beat only on the call where it just prevented a wrong booking.

6. Safety notes from the day (full disclosure)

• During tool-schema testing on a disposable copy, ElevenLabs silently re-pointed the production leasing agent at the modified test tool. Caught within minutes, restored, and production verified intact (original tool, original schema). New standing rule: no tool edits on any agent that shares tools with production — test copies get fully severed first.
• All measurement traffic was verified isolated before the phone runs continued: a dedicated test caller identity, writes confined to its own seeded test records (checked row-by-row after every call), and the booking path provably never touches AppFolio — it writes our own database and the property's Outlook calendar only.
• The tour that Fede's own test call moved (to Monday 8/24, 10 AM) was moved by the consent bug, not by measurement traffic — full RCA on file. Recommendation: leave it, cancel it with the rest of the test debris at cleanup.

7. What happens next

• The guard code exists: PR #6111 (branch only — merging it deploys nothing to any live agent, and that property is being explicitly verified against the repo's auto-sync workflows) carries the server-side consent check, the read-this-verbatim confirmation built from what was actually booked, and the "Not offered here: gym…" data injection. It is mid-review: the review bot's round-two findings (including a real case-sensitivity bug it caught in the guard's time detection) are being fixed. Held for Fede's review either way — it touches booking behavior, so no auto-merge.
• Remaining measurements before promotion: the flagship scenario at a bigger sample (target 15–20 runs) to firm up the ~75% estimate, and one missing test case (a caller who turns out to be a tenant mid-conversation) that doesn't yet exist in the suite.
• When Fede approves the guard PR: apply the tool-schema addition to the Willows lab agents only (manual step, documented in the PR — Camellia's shared tools untouched), put script v3 on the lab line with a saved snapshot for one-step rollback, and ear-test.
• Test-data cleanup owed when testing ends: cancel the two leftover test tours (Fede's moved one, and the measurement persona's) and restore the Willows line from snapshot.