Haiku FAST-tier audit

2026-08-02/03 · First deep inspection of every Haiku lane in production. 49 lanes inventoried, 44 taken through the full cycle: production-replay + synthetic corpus, judged benchmark, gap analysis, and a rewritten prompt with measured gains. ~150 agent runs, all LLM calls on subscription, prod read-only, nothing changed in the repo.

Proposed — pending review

The headline

Haiku is the right model almost everywhere — the prompts were the problem. Across 44 lanes, the audit found that the overwhelming majority of production failures were prompt defects (missing rules, ambiguous contracts, contradictory instructions), not model limits. Rewritten prompts — output contracts kept byte-compatible so no downstream code changes — closed the gaps: 40 of 44 lanes land at "keep Haiku, ship the improved prompt." On several lanes the improved Haiku prompt now beats Sonnet running the old prompt, at a third of the cost.

Typical gains on the same corpus and judges: work-order market estimate 67% → 95%. Follow-up copy 64% → 88% (the old prompt literally instructed inventing tour availability). PM commitment extraction 79% → 98.5%. Vendor call outcomes 83% → 98%. Maintenance-manual parsing 73% → 94%. Tour-reply extraction's 8 false tour confirmations at baseline → 0, with recall up. Fair-housing gate 96.9% → 99.2% with zero false blocks on real traffic.

Every improved prompt lives at /tmp/haiku-audit/<lane>/improved-prompt.md with its full evidence trail (corpus, raw benchmark rows, judge verdicts, scorecard). Nothing has been applied — each ships as its own PR with its eval as merge proof, per the decisions below.

Bugs the audit surfaced that are not prompt problems

Found while replaying production traffic — these are live code defects, independent of any prompt or model choice:

Broadcast-leak gate fails open on parse errors. The strict JSON.parse discards ~35% of correct model verdicts, and a discarded verdict is treated as "no leak." A fenced-JSON-tolerant parse is a small code fix worth more than any prompt change on that lane.
Tour-reply pipeline recorded a "running late" reply as cancelled and a reply rejecting both offered times as confirmed — two real conversations, wrong state written. Needs its own triage; the improved extractor prompt fixes the model half, not the write path.
Work-order market estimate needs one line of route code alongside its prompt (detail in its scorecard).
SMS classifier's pre-LLM regex needs narrowing — ships as one change with its prompt (detail in its scorecard).
Maestro comms compose has no safety clauses at all — no refusal rule, no scope boundary, no PII-minimisation, no output-language pin. Lowest score of any lane (71% improved). The failures were "the model did exactly what it was told."

Per-lane results

Score = judged accuracy of the improved prompt on Haiku 4.5, against gold labels + independent judges on the lane's corpus (real Camellia traffic + synthetics). Full evidence: /tmp/haiku-audit/<lane>/scorecard.md. The Sonnet arm ran as a diagnostic only — to separate "prompt weakness" from "genuine model limit" — not as an upgrade candidate.

Lane	Improved Haiku	Finding
prospect-disqualification-classifier	100%	387/387 full corpus; baseline had 13 false positives
postcall-claimed-unit-capture	99.3%	"unit as job site" confusion closed
postcall-unknown-caller-name	99.3%	speaker-attribution rule added; arms identical
fair-housing-gate	99.2%	0 false blocks on real traffic; stays fail-closed; gate itself unchanged per standing rule
scheduling-intent-classifier	99.3%	safety precedence + closing-courtesy rules added
postcall-prospect-capture	99.2%	intent-vs-identity contract fixed
vendor-message-intent-classifier	99.1%	improved Haiku beats the Sonnet diagnostic arm
sms-message-classifier	99.1%	ships with the regex narrowing as one change
tour-request-extractor	98.7%	arms tied after fix; zero disagreements
quote-parser	98.6%	arms tied 138/140
pm-commitment-extractor	98.5%	79% → 98.5%; equals the diagnostic arm
collections-legal-gate	98.4%	rubric rewrite; relevant to the collections policy slot in the outbound-safety registry
vendor-call-outcome-extractor	98.4%	fixes the live wrong-date dispatch failure mode
clara-offer-extractor	98.3%	all 16 baseline failures were prompt defects
vendor-slot-parser	98.0%	91% → 98%, 0 regressions
maestro-precheck-router	98.2%	13-point contract gap closed
tour-reply-extractor	98.0%	8 false tour confirmations → 0, recall up
prospect-name-extractor	97.9%	arms byte-identical on 62/62 after contract written down
invoice-parser	97.7%	2-point gap was a prompt defect
broadcast-leak-gate	97.7%	real fix is the fail-open parse (code, above)
wo-market-estimate	94.6%	67% → 95%; +1 line route code
maintenance-manual-parser	94.1%	73% → 94%, zero regressions
email-summarizer	93.2%	prompt worth ~11× the model delta
outbound-translate	92.8%	ship a code guard first (bare-code/punctuation inputs)
vendor-question-answerer	92.6%	"absence of a line is not evidence" rule added
reengagement-intent-gate	92.5%	we_owe_reply definition fixed
postcall-guest-card-reconcile	92.4%	236-call re-bench, judged
post-transfer-extractor	92.2%	false tour mints 7 → 0; beats the diagnostic arm
rent-roll-pdf-vision	91.9%	blank-money-cell rules added
vendor-completion-photo-validator	91.8%	small corpus (7 misses)
pm-confirmation-review-reply	90.7%	0 unearned account-access grants across all 23 traps (baseline: 7)
email-triage-3way	89.6%	improved Haiku beats old-prompt diagnostic arm
mass-comms-confirm-reply	89.4%	injection-resistance bucket widened; body-is-evidence rule
rent-roll-llm-fallback	88.6%	missing decline contract was the biggest gap
wo-judge	88.2%	all losses were HOLD failures; edit behavior perfect
knowledge-extractor	87.7%	+13.6 pts, zero regressions; corpus needs repair before more
followup-copy-generator	87.6%	64% → 88%; invented-availability instruction removed
prospect-summary-generator	87.5%	self-contradictory "non-tour closer" rule removed
turnover-charge-judge	87.2%	decision 3 below
operational-signal-extractor	82.2%	decision 3 below
email-category-classifier	77.7%	hard lane: 24% bad gold; ship gated on making `spam` recoverable
website-scraper-extractor	96.2%	fabrications 8 → 1; decision 2 below
maestro-comms-compose	71%	needs the safety clauses (code+prompt), not a tier change
vendor-completion-notes-parser	63.9%	hardest lane; billing-corruption defect live; decision 2 below

Five lanes benchmarked at baseline only

The fleet was stopped before their improve pass ran (session caps). Their baselines: par-report-vision-parser 99.4% cell accuracy · property-ticker-generator strong, keep-Haiku with one both-arms prompt defect · portfolio-insights-generator keep-Haiku (diagnostic arm was worse and broke the 7-insight contract) · tenant-summary-generator 89.4% · property-setup-chat benchmarked, unscored improve. Decision 5 below covers finishing them.

Decision 1 · Ship the 40 improved prompts?

Each is proposal-only today. Shipping = one PR per lane (or small themed batches), the lane's eval harness as merge proof, output contracts already byte-compatible.

A — Ship all 40, ordered by blast radius (gates and extractors that write state first). Recommended B — Top 10 by measured gain first, then reassess. C — Hold everything for now.

Decision 2 · The two lanes where the best Haiku prompt still trails

vendor-completion-notes-parser (63.9% vs 77.0% diagnostic — the one lane with genuine model-limit signal, plus a live billing-corruption defect on both arms) and website-scraper-extractor (96.2% vs 97.4% — marginal). Both scorecards say the prompt fix carries most of the value and should ship first regardless.

A — Ship the prompt + billing-defect fix first, re-measure, then decide the model question with fresh numbers. Recommended B — Accept Haiku as-is on both after the prompt fix; revisit only if defects recur. C — Something else (notes below).

Decision 3 · The two "needs a human call" lanes

operational-signal-extractor (82.2% improved; a 9-point residual gap on the lane that routes NTV and other operational signals) and turnover-charge-judge (87.2%; all failure classes are prompt/corpus defects but the residual matters because it gates charges). Both improved substantially; neither reached the bar the rest did.

A — Ship their improved prompts now, add a permanent eval gate, and give each a focused round-2 (corpus repair + targeted few-shots) before any further call. Recommended B — Ship the prompts and accept current accuracy. C — Hold these two entirely (notes below).

Decision 4 · Permanent eval suites

The audit built a corpus + judge harness per lane (real Camellia traffic pseudonymized before any commit, per the no-PII rule). Committing them makes every future prompt/model change on a FAST lane provable, the way the Sonnet lanes already work.

A — Commit all 44 harnesses under evals/fast-tier/ as the standing gate for FAST-lane changes. Recommended B — Only the state-writing gates and extractors; skip generators. C — Hold; keep harnesses in /tmp + S3 for now.

Decision 5 · The five unfinished lanes

Baselines exist and look healthy; only the improve pass is missing. No more heavy fleet sessions — finishing them means small, single-session runs.

A — Finish them one at a time in normal sessions, lowest-risk cadence. Recommended B — Drop; their baselines are acceptable as-is.

Method, in one paragraph

Per lane: every FAST_MODEL call site inventoried from source; corpus built from real Camellia conversations (SMS + email + call transcripts, raw — never summaries) plus synthetics (edge cases, adversarial, multilingual); baseline benchmarked with gold labels where they exist and independent rubric-anchored judges otherwise; every failure categorized (prompt weakness / ambiguous contract / bad gold / genuine model limit); a rewritten prompt with byte-compatible output contract re-benchmarked on the failure census plus a regression sample. A Sonnet 4.6 arm ran purely as a diagnostic to separate prompt weakness from model limit. All calls on subscription bearers; prod read-only; corpora with real PII stay in /tmp/haiku-audit/ and the private training-data bucket, never git.