Tour-Boundary Labeling Policy

Proposed — pending Fede's review

2026-08-20 · Blocking discovery from labeler hardening rounds 2–3 (99.9% accuracy goal). Three decisions below, each multiple-choice with a recommendation. Nothing here changes production until decided; the labeler keeps its current behavior.

Why this needs a decision

The training answers contradict each other, so no prompt can be "right." Round 3 found the gold set holds three different labels for the identical conversation shape — a lead saying "I'd like to schedule a tour" with no day or time picked yet: some are labeled leasing only, some tour only, some both. A wording fix that made the small test set 6× better made the full benchmark worse, because it satisfied one convention and violated the other two. Until one convention is chosen, the tour/leasing boundary (about half of all remaining labeler misses) cannot improve.

Context: accuracy today is 93.7% on live-traffic training data / 88.5% held-out, against the 99% minimum bar. The gold-consistency pass already fixed 6 provably wrong labels under 2 written policy rules; this page decides the rules it could not write alone.

Decision 1 — When does a conversation earn the `tour` label?

The shape in question: a lead expresses tour interest ("I'd like to schedule a tour", "can I come see it?") but no specific day/time is agreed in the thread.

Recommended
A. Intent earns the label: any caller-initiated tour request = tour (usually alongside leasing).
The label tracks what the caller is trying to do, not how far they got. Matches how the operator thinks ("who's trying to tour?"), matches the sibling adversarial fixtures we already corrected, and is the easiest rule for both the model and future labelers to apply consistently — no judgment call about how "pinned" a time is. Scheduling state (booked / not booked) already lives on the Tour row, so the label doesn't need to carry it.

B. Booking earns the label: tour only once a concrete day/time is agreed; bare interest stays leasing.
This is what the current prompt rule intended. Keeps tour high-precision ("labeled tour = a tour actually in motion"), but requires a judgment call on every borderline thread — which is exactly where both the model and the historical labelers have been inconsistent.

C. Clara-offer-taken-up earns it: tour the moment the caller engages any tour offer, even passively.
Widest net; makes almost every leasing thread also a tour thread. Cheapest to apply but dilutes the label's meaning for the operator view.

Decision 2 — Promote the corrected gold set?

The consistency pass corrected 6 of 778 labels under written policy rules (5 tour→leasing-only, 1 renewal→+maintenance), each traceable to a rule, one judge suggestion rejected and documented. The corrected file sits beside the original in S3 (camellia-gold-conversations-corrected-2026-08-20.jsonl); benchmarks currently still run against the original.

Recommended
A. Promote it as the canonical benchmark gold now. The corrections are small, documented, and policy-traceable; keeping two golds invites split-brain numbers. (If Decision 1 lands on option A, a follow-up automated pass re-adjudicates the tour-boundary slice under that rule the same way.)

B. Hold promotion until Decision 1 is made, then re-adjudicate once and promote a single combined correction. One migration instead of two; benchmark numbers stay on the old measuring stick a little longer.

Decision 3 — The noise floor vs. the 99% bar

Three benchmark runs with zero changes scored 93.1%, 93.4%, 93.9% — a ±0.8-point spread from the model's own run-to-run variance (3 of 378 conversations flip on their own). At a 99% bar, the measurement itself must be far steadier than that.

Recommended
A. Make the borderline shapes deterministic: expand the pre-rule layer (code, not model) for the recurring templates, and keep majority-of-3 voting only for genuinely ambiguous threads.
The recurring boilerplate ("just circling back…", PMS wrappers, routing menus) is exactly matchable in code — zero variance, zero cost. The model then only votes on the residual, shrinking both noise and spend.

B. Raise vote count (majority-of-5) on everything. Cuts variance by brute force; multiplies cost per conversation and still leaves a floor.

C. Accept the noise and grade on 3-run averages. Honest but slow — every iteration triples in wall-clock, and live traffic still sees single-run behavior.

What happens after the decisions

Decision	Immediate follow-through (automated)
D1 (either way)	The chosen convention goes into the labeling-policy doc + the classifier prompt in one PR; the tour-boundary gold slice is re-adjudicated by judge under it; the round-3 micro-corpus becomes its regression suite.
D2	Benchmark runner repointed at the canonical gold; PROGRESS trail records the switch date so numbers before/after are never compared naively.
D3	Pre-rule expansion ships behind the existing fail-open pattern with fail-before tests per template.