Post-transfer tour extraction: incident fix & eval proof

2026-07-29 · Fix shipped in #4861 (merged) · Eval runner in #4870 · Author: Fede + Claude

The incident (2026-07-28)

A prospect called Camellia, was transferred to a human, and agreed to a same-day tour ("maybe this afternoon… around three thirty"). The post-transfer transcript extractor — which had no idea what day the call happened — invented the date 2024-01-16. That past-dated tour was auto-confirmed and calendar-synced, but was invisible to every "upcoming tour" filter, so:

When the prospect called back from the lobby, Clara had "no tour on file" and gave no tour greeting.
The next call's confirmation sweep (keyed on tour creation recency, never tour date) texted a "confirmed for Tue, Jan 16" SMS 13 minutes after the real tour happened.

Same bug class as the July 17 email-pipeline RCA — "the AI never decides the calendar, code does" — whose doctrine never reached this extractor.

The fix (#4861, merged + prod-verified)

Anchor: the extraction prompt now states the call's weekday, date, local time, and timezone, and instructs the model to resolve every relative reference against it.
Deterministic window guard: regardless of model output, a tour date outside [today, +180d] (property timezone) can never mint a tour — code, not prompt.
Shared confirmation sender: both SMS pipelines route through one sender with a past-start guard (a tour whose start already passed can't be "confirmed"), shared dedup, and claim-release idempotency.
28 regression tests across three suites, each proven to fail on pre-fix code. Prod data corrected via the canonical reschedule path; the stranded Temporal workflow was started and its post-tour follow-up verified live.

Eval: replay of every real prod post-transfer transcript

All 17 usable human-leg transcripts in prod (of 193 rows with a transcript stamp; the rest are empty), replayed through the extraction prompt on the subscription runner, graded against hand-labeled gold. 4 are tour-relevant, 13 are negatives (vendor calls, voicemails, delinquency, availability).

Arm	Prompt	Model	Result	Notes
old-haiku	pre-#4861 (no anchor)	claude-haiku-4-5	15/17	Reproduced both prod past-date hallucinations (2024-01-16, 2024-07-19). Window guard blocks both.
new-haiku (prod)	#4861 (anchored)	claude-haiku-4-5	17/17	Both incident dates resolved correctly; zero false tours on negatives.
new-sonnet	#4861 (anchored)	claude-sonnet-4-6	17/17	Clean with the byte-exact prod prompt (see the prompt-fidelity finding below).

Findings

A second, previously unknown incident surfaced from the corpus: a Wed 2026-07-15 call agreeing to "this Friday at 10:00 AM" had auto-created a tour dated 2024-07-19. Its DDB row was already data-corrected on 2026-07-18 by the July-17 fix wave, but the extractor itself stayed broken until #4861. The old prompt reproduces the hallucination exactly on replay.
The incident class is structurally closed: past or >180-day dates cannot mint tours (deterministic guard), and past-start tours cannot trigger confirmation SMS — independent of any model behavior.
Residual risk is within-window misses (a plausible but wrong future date) — the window guard can't catch those by construction. None occurred on the byte-exact prod prompt in any arm, but the class is real (next bullet).
Prompt byte-fidelity is load-bearing: an early draft of the runner capitalized the anchor's weekday ("Wednesday" where prod emits "wednesday") and Sonnet repeatably resolved "this Friday" to a Saturday under it — a within-window miss from a one-character-class prompt difference. The review bot's drift catch corrected the Sonnet arm from 16/17 to 17/17. The runner now imports prod's date helpers and drift-checks the prompt strings before any row runs.
Model policy: both anchored tiers measure clean; any model change for this call site (e.g. the FAST→Sonnet migration, #4705) must re-run this eval rather than reason from these numbers.

Decision: no architecture change now

The proposed fold of leasing post-transfer into the maintenance-style gather/judge Temporal workflow is shelved — the simple fix is proven on real data. The ladder, if evidence ever changes:

This eval is the standing measuring stick (scripts/eval-post-transfer-extraction-subscription.ts, --mine refreshes the corpus from prod).
If within-window date misses appear: emit spans from the LLM and resolve dates deterministically via tour-date-decision.ts (small change).
Only if that fails: the gather/judge workflow fold.

Corpus is not in git (real caller PII) — mine fresh with --mine; durable copies belong in the private training-data bucket.