Clara benchmarks: results for the team

• Give the frontier model everything we hold about the person and the home, a short rule set, and trust it.
• Code owns the hard safety facts: who is writing, whether a teammate is already on it, what tools exist.
• A hold gate behind the writer catches invented facts; nothing rewrites her words.

Architecture: plan page section 2.

• Measured on 72 real production first emails. "Clean" = one home named, no counts or rosters, no re-ask, 2 to 5 sentences, nothing invented.
• Today's rewrite layers make it worse, not better: several homes or a vacancy count in 14% of drafts and 23% of sent emails. The short prompt: 0%.
• On 179 past cards the two are even (79% vs 78%); the gap there is the short prompt running long on about 1 in 4 emails.

Benchmark A, section 6. 966 cards + 110 emails; 849 of the cards were never used to tune the prompt.

• Cost per first email at list prices. Today: 106k tokens over 10+ calls, about $0.48 for the whole first-email pipeline; the writer alone $0.31.
• Speed: 22 seconds today, 7.5 with Opus, 3.2 with Sonnet.
• Still to fix before any PR: length, repeated hand-off line (3%), offering a second size, Spanish inquiries getting English tour times.

Benchmarks A and B. Sonnet 5 priced at $2 / $10 per million tokens; the repo price file says $3 / $15, which would make today 1.4x higher still.

• 475 tenant texts hit the two AppFolio inboxes between Sep 30 and Oct 23: Western Slope about 10 a day, Situs about 11.
• Clara answered one. Counting only office messages that actually answer the text (after reply pairing): Western Slope staff 100, nobody 122 of 223 (32 partly answered); Situs staff 89, nobody 152 of 252 (32 partly). AppFolio's own AI answered 11 at Situs, none at Western Slope.
• Before pairing we counted any next office message, mass texts included: staff 326, nobody 137 (superseded Oct 23 night). Calls, emails and in-person answers are invisible here.
• Topics: 246 short or unclear, 70 repairs, 42 other questions, 40 rent and portal, 26 attachment-only, 20 lease and move-out, 20 thanks, 11 confirming a visit.

Benchmark C, section 6; reply pairing from Benchmark D2, section 8.

1,994 real tenant texts (recent inbox at both companies + Situs 2018 to 2026 history), sorted by Opus 5.5

• Almost half of tenant texts need no reply at all: thanks, reactions, answers to mass texts, closed threads.
• Of the rest, about a quarter are routine today, and another third become routine once Clara can see the right data.
• The pattern is the same at both companies and across five years of Situs history.

Benchmark D, section 8.

• Share of the 396 "Clara could, with more" texts. One tool covers a third of them: knowing when the vendor is coming and where the work order stands.
• Where staff beat Clara in the head-to-head, it was always something newer than our records: a plumber running late, a repair pushed back, an expired portal invite.

Benchmark D, section 8.

• When she writes, the judge preferred her over staff 24 to 2. Nothing invented, no "let me check with the team", no promises she cannot keep.
• She replied to only 18 of the 50 texts she herself called routine: the decision prompt is too cautious, the model is not.
• The one real risk is talking over a teammate. That becomes a code rule, not a prompt line.
• Sonnet 5.5: half the price, twice as fast, 46% more team items, and the judge preferred staff 6 of 14 times on the Situs history.

Benchmark D, section 8. Superseded Oct 23 night by Benchmark D2 (next slide): 5 of 18 Opus judged pairs were against a message that was not an answer. About 31 judged pairs per arm, so a few flips move the picture.

• Zero replies on threads an office teammate is running. Was 10 to 13 per set.
• Zero invented facts, no "check with the team", no promises outside her actions.
• Open problem: too quiet on routine asks, 10 of 50 on recent texts. Fewer replies is the goal; this drop is not. Next: loosen the prompt there.

Benchmark D2 and D3, section 8. D3 swaps the 24 h teammate gate for a model decision; D2 figures above are superseded; its one miss (#4) needs Fede's re-grade; in-sample. Two outcomes only (reply or silent), complete context, teammate rule in code, staff replies paired only when they answer. Judged pairs after pairing are small (Opus recent 6 / 0 / 0, n=6).

• When she files, it is right: same issue, location and urgency 8 of 8; permission to enter 7 of 8; no duplicates.
• She stays out of staff-run threads: 140 of 141. Where staff ran the thread she filed 1 of 19, correctly.
• Open problem: where nobody runs the thread she still misses 16 of 23, mostly silent on a text addressed to a staff member by name or a repair already in the portal.
• Sonnet 5.5 filed 11 of 42 but chimed in on 11 staff-run threads.

Benchmark G2, section 8. G numbers ran under the old three outcomes and are superseded. 250 of 261 replays are one turn; the staff-run label (one Fable read, 143 of 261) is generous; 42 episodes are co-tenant duplicates. 28 episodes on the review page.

• Rule since Oct 23: Clara speaks only to a positively identified prospect, or a tenant where that company's tenant switch is on. Unknown = silent. A former tenant is a tenant.
• *The one model "leak" was a bare "Yes" from someone our records called a prospect at the time; the tenant label came later.
• Deciding costs about a cent per message with Opus; the role lookup is free.

Benchmark C, section 6; role layer in section 9.

• Nothing invented, no "check with the team", no promises, in any of the 78 Spanish replies.
• Prospect emails: 3 of 4 Opus drafts say "el Friday, October 23 at 3:00 PM" because tour slots reach the writer as English text. Easy fix, must be proven on the 4 Spanish cards.
• Knowledge entries come out translated word for word. The fix is in the writer (say it in natural Spanish), never Spanish entries. Register slips to usted where the office writes tú.
• The office itself answers Spanish texts in Spanish 221 of 270 times.

Benchmark F, section 8. Tenant numbers superseded Oct 23 night by Benchmark D2: Opus 15 replies, all in Spanish, naturalness 4.6, 0 knowledge entries translated word for word. Judge wrote and read in Spanish; no native speaker has checked the scores yet.

• Prospects: fix the four found defects, then the short prompt replaces today's stack at every property, no switch.
• Tenants: two outcomes, reply or silent; teammate rule now in code (D2). Loosen the "reply" bar on routine asks (10 of 50), then hold every reply for a supervised week at one company.
• Spanish: tour slots as dates not English strings, register from the thread, no Spanish entries, the writer rewrites.
• Tools: vendor timing and work-order status first; it unlocks a third of the texts Clara cannot answer yet.
• Open decisions are listed on the plan page, section 1 and section 8.

All numbers: clara-lane-order-2026-10.