Classifier Rebuild Plan

Shipped and live: #22879 (money owed holds when Collections is Off or Supervised), #22880 (its eval), #22972 (a general reply follows who is writing), #22995 (an unknown sender's reply waits under the first module that isn't Off), #23220, #23237, #23258 (the hold says "We don't know who this person is"), #22935 (Resident Services is a real module), #23380 (Resident Services is its own reply topic and sends only while Live), #23428 (silent hand-off, slice 1).

In flight: #23500 and #23508 (silent hand-off 2a and 2b: a reply that is only a hand-off or a hedge sends nothing), and the neighbor-complaint hold (being built, no PR yet).

Still true on main:

• The reply check (classify-reply-topic.ts) has five labels (renewal, maintenance, collections, resident services, other). It runs on Haiku 4.5 with a 6-second limit. On a timeout or error the renewal hold reads it as "other", and the Supervised hold falls back to the thread's module, so most replies still go.
• shouldSpeak returns "silent" only when a teammate already holds the thread, or from the silencer that runs before a draft. Never because of the kind of message, the topic or the risk.
• The page labeller is separate: 8 topics, reads the whole thread, Haiku 4.5, lands about 10 seconds after the reply decision.
• Preview mode is live on main. The "would-have" records it is meant to keep (the draft Clara would have sent) are not on main yet; the maintenance session owns them.

344 of the 639 moments are texts and emails (179 + 165); the other 295 are voice, which this plan leaves alone. Every replay gate below uses the 344. A second, separate measure of page labels: the 10-27 audit hand-judged 248 messages at 84.7% strict and 92.3% counting defensible calls. This plan's page target uses the turn-study measure: 75.7% today, 90% target.

Reviewer agreement (Fleiss κ): who's writing 0.85, kind 0.70, topic 0.68 (0.82 when all three named one), action 0.67, any risk 0.39.

Fede's engine: gather context → shouldSpeak → writer → draft check → record → proof rail. Tenant comms is Gera's swarm. The core doors (shouldSpeak, the notice engine, the send door, the writer) are Fede's.

Value rule (Gera): reply only with real value. A greeting gets a greeting. Never "I'll let the team know". A hand-off is silent. This covers texts and email; voice and the tour offer keep their wording. The draft check's "adds value" question generalizes the silent hand-off work (#23428, #23500, #23508): those stay as the fixed rule underneath it, and the check catches the hedges they miss.

No topic: the reply follows who's writing. Prospect → Leasing, resident → Resident Services, vendor → Maintenance, unknown → held under the first module that isn't Off.

The whitelist: every topic in the message and in the draft has to resolve through one map. Live → can send. Supervised → a person approves. Off → silent. general or none → use who's writing (above). Truly unmapped → silent. Nothing sends by being left out.

One TurnAssessment per inbound text or email. Records are only ever added, never edited. The message's page topic and subtopic stay where they are.

Each record also keeps what the labels were when the decision was made next to the current best guess. If a later relabel would have changed a past decision, it goes to review. It also stores the model, prompt, policy, keyword-list and taxonomy versions.

• Inputs: the inbound message and the conversation before it (only what existed then), who's writing, thread state, each module's state at the building, the draft, the tool results the writer cited, and the policy table (topic → module, the value rule). It does not see the writer's reasoning or confidence.
• Outputs: allowed (with topics and the module each maps to), adds_value (with why), true (with any unsupported claims), risk flags, kind. One call returns all three verdicts (Fable: three calls to the same model triple the cost and add no independence). A missing verdict counts as a fail.
• Writer's own read: recorded but never used to send. One exception: its risk flags join the "any source" rule, because they can only make Clara quieter.
• Time: text and email only. Opus 5.5 on low effort to start. Added time p50 ≤ 10 s, p90 ≤ 20 s. These are bench targets, not promises: the audit estimates 8 to 15 s per call. Hard limit 30 s, after which it's silent and an engineering alert fires. Alarm if the daily timeout rate passes 2%.
• Cost: about two cents a turn at list price (estimated).

• Sources: the 344 text and email moments from the study (relabelled blind), a keyword scan of production for money cases, Preview "would-have" drafts once they land, shadow drafts, and the 40 cases on the scenario board.
• Each row gets: the right action (reply, short reply, hold, silent, urgent), the five dimensions, and for draft rows, each arm's draft graded allowed / adds value / true.
• Who labels: Fable and Astra label independently, without seeing each other's labels or any model output. Gera and Fede settle disagreements on the scenario board. What staff actually did (replied within 48 hours, approved or rejected a hold, a Preview draft) flags rows for a second look; it is not the answer.
• Agreement bar (κ): reply or not ≥ 0.8; must-hold ≥ 0.8 and ≥ 95% raw agreement; topic ≥ 0.7; kind ≥ 0.7; who's writing ≥ 0.85; each risk flag ≥ 0.7, measured per flag and never pooled. Risk is 0.39 today, so: a 100-row pilot first, rewrite the guidelines until the pilot clears the bars, then label the rest. A safety case still unsettled after that is marked "hold".
• No leaks: labellers see only what existed at that moment. Splits are by conversation, customer and time: development 550, calibration 300, sealed 500. Sealed is the most recent slice plus one whole customer held out. The 300 locked money rows run once, at release, and their text is never shown to whoever writes prompts. Fixes go into development. Sealed is refreshed monthly. Real text stays off git, masked, like Clara Bench. Fixtures in the repo use synthetic ids only.

Primary: the unsafe-send rate among sends: sends whose right answer was "don't reply", or that fail allowed or true, divided by all sends. We report it with a one-sided 95% upper bound, overall and per module.

Always shown beside it, so "hold everything" can't win: missed holds on the 300 locked money rows (0 of 300 means under about 1%), and coverage, meaning the share of right-answer-is-reply rows that actually send.

Secondary: blocks per check; value violations (a hand-off or hedge that went out); page topic agreement (75.7% → 90%); Resident Services vs Collections mix-ups; time p50 and p90 and the timeout rate; how often a label flips across 5 repeats of the hard rows; cost per 1,000 turns; holds per customer per day; digest size.

Honest sizing: the key is for design and catching regressions. It cannot show a 0.1% error rate; that needs about 3,000 graded sends and builds up after launch. It is never a launch gate.

1. Shadow. The whole new pipeline runs beside today's: the new shouldSpeak, writer, draft check and combine step. It makes a would-have decision on every text and email turn, including turns today's path holds or silences. That second half needs the Preview "would-have" records, a hard dependency owned by the maintenance session. Every disagreement between old and new gets graded in both directions, plus 10% of the agreements. At least two weeks, through Nov 1, with at least one of those weeks on the writer version that will be live when we switch.
2. One customer. A founder's call, Fede or Gera. It uses Fede's release-template company setting, defaulting to shadow; Fable treats this as an extension of the template's "new writer" carve-out, to be approved alongside D1. Blocks are reviewed daily for a week on the block-count page.
3. Everyone. Then the setting and the old reply check are deleted (D4).

Rollback: flip the setting back. The check can only block, so rolling back makes Clara louder and never opens a new way to send. A timeout never goes back to meaning "send".

One taxonomy module (the topic list, definitions, the topic + subtopic → module map, the risk flags) is read by both the page labeller and the draft check. The old reply-topic classifier is retired into the draft check. The page labeller stays separate and never decides a reply. When the draft check and the page disagree on the same message, that is logged as free test data.

Page chips: Haiku labels within the 20-second bar, then an Opus 5.5 pass relabels inside the existing stamp workflow through the one canonical writer (never a new classification caller, and a relabel never reverts to "Classifying"), on untagged and reconsidered rows only; the record keeps both. Fable's pick: the only option that keeps Opus as the standard and the 20-second bar.

Each one is ≤ 300 lines, branched from main, and works on its own. Order corrected by Fable: thread state is recorded before anything reads it, the combine step gets its own slice, and the block-count page lands before any customer is switched on.

• Slower texts: the check adds 10 to 20 s. Accepted for text and email; voice is untouched.
• Too quiet: mitigated by the coverage bar, the silence budget, and shadow that measures both directions.
• Same-family blind spots: the writer and the check are both Opus. Mitigated by the keyword floor, the check's different job, and keeping it blind to the writer's reasoning.
• Weak risk labels (κ 0.39): the pilot and guideline rewrite come before the full round.
• Preview records slip: then shadow measures only one direction, and D3 waits.
• Fede's writer changes during shadow: at least one shadow week must be on the writer that will be live.
• Rent-cycle drift around the 1st; instructions hidden in a tenant's message; an Opus outage makes everything quiet (fail closed, plus an alert).

1. Gera: the edge between Resident Services and Collections. Proposed: plain balance, due date, autopay and portal questions are Resident Services. Owing, late fees, a promise to pay ("I'll pay Friday"), "am I behind?" and "can you waive the late fee?" are Collections.
2. Gera and Fede: the silence budget. How many quiet turns per customer per day, and how big a digest, is acceptable? Needed before D2 can be judged.
3. Gera and Fede: money trouble that goes quiet ("I can't pay until the 14th"). Is tomorrow's digest fast enough, or does the owner get a same-day notice?
4. Gera and Fede: a tenant who follows a quiet turn with "hello?? anyone there?". Does that get a greeting back, or a second, escalating tracked ask?
5. Gera: Friday night. Is one "the office will reply Monday" ever allowed, or is it always silence?
6. A founder (Fede or Gera): which customer enforces first.

Settled here, not open: an Off module stays silent and goes to the digest (Gera's value rule already decides it: "we don't handle that here" adds nothing). Collections is never Live until counsel answers; the action is sending counsel the per-customer question now. The seam for the draft check is an engineering sign-off for Fede on D1, not a product call.

• Gate on a model's stated confidence, or let a majority vote erase a risk flag.
• Add another place in the code that classifies on its own.
• Leave a switch behind after rollout. The rollout setting is deleted in D4.
• Send "I'll let the team know", or put a failed draft in front of a person.
• Touch voice in this plan, or let a reply decision read the page label.
• Rewrite drafts. Checks only block.
• Train our own model before 5,000 to 10,000 reviewed labels exist. The records are designed for it from day one.
• Put real customer ids in fixtures, or delete the old check before two clean weeks.

• Added the whitelist slice (B1b). v1 stated the rule but never scheduled it, which left the "other sends" leak open for weeks.
• A failed draft is never held. v1 sent failed drafts to a person when the module was Supervised. Now: silent plus digest, and a hold only for drafts that passed everything. Two of three holds already expire unseen.
• Shadow measures both directions. v1's shadow could only see turns today's path drafts, so it could not measure going too quiet. Preview records are now a hard dependency.
• Split A1: the fail-closed flip ships alone, now; the Opus move comes after the baseline. The send-door files are Fede's.
• Bigger money slice with a locked test (450, 300 locked), a 100-row pilot before the full labelling round, risk agreement measured per flag.
• Decided the technical forks: fast label then Opus relabel for page chips; one draft-check call; "a teammate is active" means newer than Clara's last message and under 24 hours, not minutes; the time budget; the splits and bars; the writer's read is recorded but doesn't gate.
• Fixed facts: 344, not 639, moments are in scope. Two page-accuracy numbers were being mixed. v1's p50 ≤ 8 s target was below the audit's own estimate. Section 2 and section 5 contradicted each other on Supervised.
• Moved the decisions: "Off module" and "Collections until counsel" are settled, not open. Added the silence budget, the money-notice speed and the repeat-ask calls.