Clara Bench: one repeatable test for model upgrades and big refactors

Where the drafts land: three options, one recommendation

Decided (Fede by phone, Oct 23 08:07): shadow mode is on HOLD until Fede has read this page. When it goes live, drafts go to the property's Slack channel using the existing supervised-mode card (option A), not a local page and not a new channel. What that needs before the first run: the supervised card gets a "shadow" variant with no approve button (an approve that sends is not shadow), the four reactions are read back as labels, and the 24-hour expiry is irrelevant because nothing is waiting to send. That is a small change in the app, so it ships as one small dark PR after Fede's go, not tonight.

The micro-feedback loop is the product of the weekend

Every draft gets at most one tap and an optional note. The four taps map straight onto the scoring layers in section 4 and the buckets in section 12, so each tap is a labeled benchmark item, not a comment to read later:

A label record is: message id, customer, audience, the bucket chosen, the draft's hash, the tap, the note, who, when. It is appended to one labels file in the corpus bucket beside the frozen set. After about 100 taps the loop closes: the judge is calibrated on them (section 4, decision D5), the bucket rule is fixed on the misses, the same messages are re-run, and the before and after are compared on the labels, not on opinion. That is also the moment the Bench set's triage surface (300 items) gets its first real labels instead of a grader's guesses.

Bar and stop rule for shadow mode

• Runs on: every inbound Situs and Western Slope inbox message, with the thread before it and the production wake rule applied first, so the page shows "would not have woken: staff spoke last" rather than a draft nobody would have sent.
• Volume: about 30 messages a day across both customers (202 bursts in 7 days last week), so a weekend is about 60 to 90 drafts, under an hour of taps.
• Graduates to the next rung when: 100 labels, "should not have replied" under 5%, "wrong bucket" under 10%, zero invented facts in the Good pile. The next rung is supervised (the same card, with approve), which Western Slope asked for on Oct 14; "held when unsure" comes after it (ladder in section 12). Each rung is Fede's turn-on, per customer.
• Stops when: any run writes anywhere but its drafts and labels, or the drafts stop matching the inbox copy. Default off per customer; Fede's go starts it, and his go is pending his read of this page.

Prototype run tonight

To test the claim that this can run now, a builder ran it tonight on the 10 newest inbound messages in the two inbox copies (5 Situs, 5 Western Slope), each with its thread, through the production wake rule and Clara's real text prompt, on the subscription, with every write path removed. Nothing was sent. The code sits on a local branch in a worktree, committed locally, no push, no PR, no CI.

What the run taught, in one evening:

• The wake rule has last night's "tenants treated like prospects" problem built in. A thread counts as a tenant thread only if every party is a tenant. A renter who once applied through AppFolio also carries applicant and prospect records, so 5 of 5 Western Slope threads and 2 of 5 Situs threads counted as prospect threads. Western Slope's inbox door is on, so on those threads the only thing keeping Clara quiet today is that a staff member spoke last. A renter's first text on a thread with no office reply yet would wake her as a prospect. If the party types in the inbox copy match what the filing step sees, the fix is one line: any tenant on the thread makes it a tenant thread. Unverified tonight; it is the first regression case the labels should confirm.
• This sample never tested the case that matters. All 10 threads had staff on them, so Clara stayed out, correctly. The daily job must lean its pick toward threads where the office has not spoken yet, which is where a draft would actually have gone out.
• Two of the "tenant messages" were not from tenants (a vendor, a staff text). The picker needs a "the renter wrote this" check; that is the deterministic pre-layer in the diagram doing its job.
• Clara saw no knowledge base and no lease facts, only the thread and the company name. Draft 1 offered the leasing phone line for a rent-credit dispute; the number was real (from a mass text in the thread) but wrong for the question. The weekend job should add the property's knowledge facts, which the inbox replay last night also lacked.
• The draft model was the code default (Sonnet 4.6), not the production writer. The job takes a model override; shadow must run the exact production model or its drafts are not what Clara would have said.

To run it daily for a week, the way Fede chose: the shadow job runs where supervised mode already runs (the app, not a Mac mini cron), on every inbound inbox message with the thread before it, leaning to threads where the office has not spoken yet, on the production model, with the property's knowledge facts attached, with the write-free guard and before/after proof on every run. Each draft posts as a shadow card in the property's Slack channel: no approve button, four reactions as the labels, a thread reply as the note. The label export feeds the labels file in the corpus bucket, which is where section 3's triage items get their first human labels. That is one small dark PR after Fede's go; nothing runs tonight. The local prototype stays what it is: proof that the path is write-free and that the drafts can be read.

What we have already learned the hard way

Fede asked for another pass over the retros of the experiments that missed. The pattern he suspected holds, with one correction: the biggest single cause at Camellia was not too little data. It was that every alert went to our own inbox and the Camellia team answered 5 of 144 emails. The retro's words: "we built a coworker who reports to an empty desk." Shadow mode's desk is Fede's, which is the one desk whose reply rate we can measure this weekend.

Sources: co-worker release hub (Sep 3 and Sep 4 sections), staff notification architecture, bake gate, feature-flag audit, staff loop v3, Sonnet 5.5 release page, voice bake-off, Constitution preamble. One standing ruling to reconcile: on Oct 16 Fede said "watch mode no, do not build new things." This section respects it by building nothing in the product: shadow mode is a local job reading the inbox copy, the same thing last night's replay already did, plus a page with four buttons. The Oct 23 steer ("something we could turn on over the weekend, never sending anything") is the newer ruling and wins.

Rules we already wrote and keep breaking (each one broken by at least two rows above): a customer who asked and a reviewer who answers (Constitution I.2, I.4); delete before you add, one mechanism per signal (II.2, II.3); real history is the judge (III.2, III.6, testing playbook); the bench is not the customer (III.3); a proof must be able to fail (III.8); a switch only for a new capability (IV.3); and one that is still not a standing rule anywhere: agree the bar and the budget before the run. Decision D7 (section 9) makes it one.

Not needed for shadow mode (the "later" list)

Precedents that worked: the voice bake-off (866 calls, bar agreed first: "match production on every rule", decided in three days) and the effort replay (137 real conversations with a same-config control, so the diff had a noise floor). Source for last time: prompt architecture page, sections "New first-reply prompt: results" and "Cost".

Every item is a real past moment with its real history up to that point. Clara writes the next message only; the judge reads it against the real facts of that day (homes, prices, calendar, knowledge base), which are frozen with the item. Multi-turn drift is not replayed: that proof stays at the Willows.

Frozen means: a manifest of item ids with a content hash, in git; the items and their day-of facts in the existing guest-card-corpus bucket; the baseline's outputs captured once ("Baseline Oct 2026", Sonnet 5 on today's prompt) and never regenerated. A set is versioned (v1, v1.1) and only ever grows: a refresh adds the last quarter's items, it never swaps old ones, so two runs a year apart still compare on the same 900. A build check fails if any item's text appears inside a Clara prompt.

Fede named the three layers: how well Clara talks, what she must never do, and whether the gates catch what matters. They are scored separately and rolled into one number per candidate.

The one percent. An item passes when it breaks no layer-B rule and the judge rates it a tie or a win against the baseline. The Clara score is the share of items that pass, with its error bar, beside the same number for the A/A run. Layer C is reported as two rates (catch rate on planted defects, false-hold rate on the clean slice) and never averaged in.

The bar, written before the run: the candidate's score is not more than 5 points below the baseline at the lower edge of its error bar; the gap must exceed the A/A spread; zero layer-B violations on the regression cases; refusal rate not above the baseline; latency and cost reported in absolute numbers. "No worse than today" is the bar, because Fede's own baseline is "99% right".

The judge. One model, not a panel, and never one of the candidates (self-preference is measured at 10–25%). Calibrated before its first counted verdict: 100 items labeled by Fede and Gera (30 is the floor for a smoke tier), target 85% agreement on non-tie items, catch rates for passes and fails reported separately. Both answer orders are judged and a win counts only if both agree. The legacy EVAL_JUDGE_MODEL pin (Sonnet 4.6) stays for the existing nightly evals; changing it re-baselines those scores, and the Bench does not need it.

Staff replies are context, not the yardstick (Fede, Oct 24, by voice): The shadow feed of tenant texts is a source of episodes only. What the office actually replied is shown as one optional column when a staff reply exists and is visible, never as the pass or fail bar, because staff may not answer, may answer outside what we can see, and vary a lot. The primary grade is a numbered review set graded directly by Fede or the engineering team against his rulings, with volunteer staff added as graders when he asks for them.

• Subscription only. Metered keys never; the existing -subscription runners and the bearer client are the pattern. Seven concurrent sessions hit 429s last time, so the runner caps itself at four workers.
• Stop rules. A run past its clock writes a verdict of "incomplete" with every number so far and the reason; a run that cannot assert the served model on every call is a harness error, not a score; plumbing failures (timeouts, 429, cut-off at max tokens) go to a sidecar, never into the percent.
• Sized before spending. 300 paired items detect a 5-point margin at 90–95% confidence. Detecting a 2-point gap would take 600–1,500 items and is out of budget on purpose; the bar is "no worse than 5 points", not "better".
• Each case runs twice. One pass is a point estimate with no error bar; the Sierra benchmark showed single-try pass rates overstating "passes every time" by 60%.

The voice bake-off's lesson, in Fede's words: "maybe there's a migration process when switching, not just flip on." Anthropic's own guide says the same: port the request, remove old workarounds, re-sweep effort, and only then judge. For the writer today (adaptive thinking, effort high, 16,384 max tokens, no forced tool choice on the first call) the pass is short:

One process for every prompt or model change to Clara, built from the tools we already run. Most of the work is wiring them in order, not building new ones. Every change walks the same six stages on the testing playbook ladder; four pieces are missing, all small or medium.

What we already have

Sixteen pieces already exist. Rung numbers are from the testing playbook. Rung 1 and 2 items are build hygiene: they run in CI and catch breakage, but they are never the proof.

Stages, what we have, what is missing

Exit bar, written once

Adds to the score bar in section 4; it does not replace it.

Worked example: the first email to a prospect

The next model release is a rerun: the migration pass, then one new arm on the same replay set, judge, review set and exit bar, read on the verdict screen. If it passes, it walks stages 4 to 6 like any other change.

Sources: Clara lane page sections 0, 0c, 6, 12; testing playbook; always shipping (dark-features rule); feature flag audit. Folded in from the retired page "Clara changes: how we benchmark and release, with what we already have".

Every run is a row on a shared site, like the CI/CD and ingest apps Gera built, so a model upgrade and a refactor are read the same way: before and after, on the same set.

Build shape (not tonight): the runner writes one JSON per run to the training-data bucket; a static site reads the bucket, like the docs site reads artifacts. No database, no login beyond the shared link. The same runner serves refactor PRs: a PR that touches the writer path adds a "bench: smoke" label and the smoke tier posts its row.

Fede's ask: look at everything between the first inbound turn and the send, get rid of the prompt rewrites, keep one last gate that asks "are we about to say something stupid, illegal, or leak information". Read-only on main after the Oct 23 merge of #21992, which already deleted four dead rewrite steps.

What a held message should be. One surface: the supervised post, with the draft, the named reason and approve/reject, in the property's channel. The mechanism exists (holdEscalationEmail); the gate-held first email just does not call it. The rule that "a blocked draft must never reach anyone" was written for fair-housing holds, which no longer hold. Open: why 53 of 61 Western Slope resident texts expired unreviewed; who owns the expiry.

Fede's frame (Oct 23): every message on a customer's shared AppFolio inbox lands in one of three buckets, and the buckets are the product. Design only: nothing here is built or turned on.

What tenants get today

Clara wakes on no tenant text on the AppFolio inbox unless the tenant door is on (wake-on-reply.ts:7-13). A tenant who reaches her gets this (main, 7e6bd5ecd3):

The three buckets as the product

Gera's 90-day Western Slope maintenance audit (Oct 21) already sorted texts this way: some needed no reply ("just a simple, okay, thing"), 23 needed a person ("billing, deposits, rent timings"), 36 Clara could handle ("is the plumber coming?") (transcript).

The Oct 22 inbox replay (202 bursts, 7 days, nothing sent) against the buckets. The harness had the wake rule off and showed no earlier thread, so these are upper bounds, not production rates; the 52 and 61 may overlap.

Decision W1: how Clara decides to wake up

Kept, not rebuilt: the wake rule's nine no-wake reasons (wake-on-reply.ts:24-33); the staff hold, already read at the send door (auto-reply-hold.ts; dispatch-envelope.ts:464); the automated-sender gate (automated-sender-gate.ts:95-124), its idea extended to texts via the inbox copy's bulk and notice labels.

Are today's Haiku layers a good idea? For a narrow question they can answer, yes: email triage calls automated notices no-reply 33 of 33 times at Western Slope (automated-sender-gate.ts:14-16). For "should Clara speak", no: Haiku cannot see the work order, the balance or who spoke last. Keep email triage, limit the tour classifier to prospects and applicants (inbound-router.ts:1440-1445), add none.

Decision P1: does Clara's personality change by audience?

Vendors. The in-house handyman and unknown vendors get prospect rules, as the code's own comment predicts (clara-unified.ts:69-72) and 9 of 9 handyman traces show (prompt-architecture page, bug fix 0). Fix: a vendor sender type beside tenant, prospect and unknown (resolve.ts:206), from the PMS adapter's neutral vendor record, plus a vendor job block.

Decision O1: the open-items layer (name TBD; not the promise ledger)

Concept. A bucket-2 message with no human reply for 24 hours goes on one morning list: a link to the thread, why it needs a person, a suggested next step (section 13). Default off per customer. No nudges to tenants. No urgency tiers in v1 ("first get the mechanism", Fede, Oct 23).

Decision H1: human ↔ Clara handoff

Clara to human. Bucket 2 is silent: the thread is the record, the digest the reminder, the ping for urgent items (C1). Staff never tell Clara anything; their message on the thread is the signal. That is the customer's "very deliberate… delineated" handoff with zero steps for staff.

Human to Clara. Today one live staff text ends Clara's part in the thread for good: a hold nothing clears (auto-reply-hold.ts:44-65; conversation.ts:2479, "sole writer"), plus the wake rule's own stop (wake-on-reply.ts:85), two mechanisms on one signal (II.3). Letting her answer again when the renter asks something new she can do would cut against Fede's "IF another human replies we stop" (Sep 3) and his Oct 16 ruling of no clock and no come-back (wake-on-reply.ts:13-15). OPEN, Fede. For v1 keep the stop; shadow tags the would-resume cases so the reactions decide.

Decision C1: what Clara does when she cannot help (co-worker mode)

Fede's caveat (the team "may not actually see the message"): both happen. 61 of 202 replay messages also drew an office or AppFolio AI reply, yet the customer said "Probably the only person getting a notification about the response is me". Silence covers the first case, the digest the second, the ping the urgent few. Fede's own test stands: a new hire whose every answer is "let me check" gets fired.

The confidence ladder

Tenant comms climb; they do not jump. Leasing was turned on and watched ("leasing, we went YOLO"); for tenants Fede wants review first, and so does the customer.

Benchmark cases: bucket accuracy

Challenges, answered or open

What this deletes or folds

Constitution II.2. Changes apply to the shared inbox only; Clara's own line keeps its deferral sentences until that lane decides. Fixes ship with no switch; Clara on the tenant inbox is a new capability, default off per customer, Fede's turn-on.

Shadow is on hold until Fede has read this page (phone, Oct 23); "yes" means needed for its first run.

Fede's ask (Oct 23): "there must be guidance, proven patterns out there… source beyond the big players." Seven research lanes ran in parallel on Haiku, about 130 pages, primary sources preferred. Honest coverage note: many pages were blocked (G2, Capterra, Reddit, Salesforce, several vendor docs), most vendor pages are marketing, and the field documents escalation far more than silence. Rows marked "snippet" come from search results, not a page we read. Raw citations with their verification level are saved beside the brief in sources-ai-employee-patterns.md.

Where the field disagrees, one line each side: Intercom and Twig escalate on written rules, Lorikeet and Gradient Labs on the model's own confidence plus checks; we take rules first, confidence second, because the labels will tell us where the band sits. Some ship LLM judges in production, Yan and Shankar say human grades are still needed; we calibrate the judge on Fede's taps before any verdict counts. Several vendors run offline simulations, Intercom says only live A/B counts; we do both, in that order.

What this changed on the page: bucket 2 is defined by case type before confidence (W1, C1); kappa is reported beside agreement in section 4; "should not have replied" is scored per candidate model; every shadow draft carries the property's knowledge facts; the never-do list gains "never claim to be a person" and "a request for a person is bucket 2 at once." Nothing we found argues against shadow first or against the four-tap feedback loop; the field's one consistent warning is to never go live on everything at once.

Main loop on Fable 5.1; all model calls on the subscription, no metered keys. Compare with the run this page replaces: 42 hours and 146 subagents for no verdict. The design cost about a twenty-fifth of the wall time and a seventh of the subagents, and the next step it proposes (shadow mode) costs about four seconds of model time per ten drafts.