Measured answer to Fede's question "I would not mind some more latency for higher quality": Clara's email/SMS drafting on Sonnet 5 at effort medium (today's prod) vs high vs xhigh.
2026-08-27 · DECIDED 2026-08-28: Option A — shipped and live-verified (see banner)
High won, narrowly. Extra-high lost. Raising effort one rung (medium → high) fixed 2 of medium's 5 graded failures and broke nothing, at ~0.7s more typical latency. Two rungs (xhigh) fixed the same rows but dropped a tour invite on another and made the slowest reply 4× slower. The edge for high is consistent but small-sample — the decision options are below.
| Effort level | Leasing replies, LLM-graded (37 email+SMS rows) | Renewal SMS, deterministic (19 rows) | Typical latency (p50) | Slowest (p95) | Truncated / empty |
|---|---|---|---|---|---|
medium — prod today | 32/37 (86%) | 19/19 | 2.3s | 4.4s | 0 / 0 |
high | 34/37 (92%) | 19/19 | 3.0s | 7.2s | 0 / 0 |
xhigh | 33/37 (89%) | 19/19 | 5.2s | 17.8s | 0 / 0 |
Same 37 leasing rows and 19 renewal rows per level (deterministic sample, every 3rd row per channel bucket), one draw per row, model claude-sonnet-5, adaptive thinking, max_tokens 8192, judge pinned at claude-sonnet-4-6 / temperature 0. Voice rows excluded — voice doesn't run this tier.
| Scenario | medium | high | xhigh | What happened |
|---|---|---|---|---|
| SMS — per-unit specials: lead with the units that have the deal | fail | pass | pass | Medium got the price range right but ordered the units wrong; both higher levels prioritized correctly. |
| Email — single matching unit: quote rent/size but don't reveal the unit number | fail | pass | pass | Medium leaked the unit number ("304"); both higher levels held it back. |
| SMS — only two 1BRs left: honest availability, no scarcity bait, end with a tour push | pass | pass | fail | xhigh wrote a good reply but ended without inviting a tour — the one hard requirement it skipped. (Clara's leasing doctrine is tour-first, so this miss matters.) |
No scenario favored medium over high. Three further rows failed at every level (an amenity-hallucination rubric, a missing-concessions-special case, and one single-unit email rule) — those are prompt-level issues that more thinking does not fix, and they're the natural targets of the separate Sonnet-5 prompt-audit work.
high (recommended)One-line change plus its unit test. Fits the stated appetite ("more latency for higher quality"): +0.7s typical, +2.8s worst-case, both flips in its favor, nothing regressed, renewal flows untouched. Caveat honestly: 2 flips at n=37 is a consistent direction, not statistical proof.
mediumDefensible if the bar is "change prod only on statistical proof." Keeps today's latency; leaves the two graded failures (unit-number leak, specials ordering) as-is.
All 109 email/SMS leasing rows × 5 repeats per row, medium vs high only, judged offline. Now cheap to run (the grading bottleneck below is fixed) — roughly an evening of background subscription time. Turns "consistent direction" into a real verdict before touching prod.
A and C combine naturally: ship A now, let C confirm or revert it.
scripts/eval-reason-first-drafting-ab-subscription.ts (the #6011 proof harness) extended with --effort / --stride; offline grader scripts/eval-reason-first-judge-offline.ts. Worktree propflowai-effort-sweep, branch fede/effort-sweep-email-sms.claude-sonnet-5, thinking adaptive, effort as labeled, max_tokens 8192, temperature omitted.evals/datasets/leasing-response-quality.yaml (email+SMS rows) and evals/datasets/renewals-sms.yaml; raw per-row JSON (incl. sampled indices and full reply text) in the session scratchpad effort-sweep/ directory.