Tour Decider v2 vs Legacy — Camellia Replay

• The full conversation history: all 791 real Camellia conversations pulled from production, read-only (620 phone calls, 91 emails, 80 texts) — 9,271 individual messages, 3,471 of them from prospects. 135 of those conversations actually touch a tour (propose/confirm/cancel/decline/reschedule) — 119 of those could be replayed turn-by-turn against reconstructed state; the other 468 non-tour conversations went through as a coverage pass (nothing to replay, but nothing skipped either — every single one is accounted for).
• 43 previously known bugs — every tour/booking bug found since incidents in July and August, turned into a test scenario with the state it started in, what was said, and what should have happened.
• 124 adversarial scenarios across 23 categories — deliberately tricky replies (ambiguous "sure?", stale replies to superseded proposals, two tours in flight, a property manager's own message landing on the thread, "STOP" mid-flow, prompt-injection attempts aimed at the AI classifier, and more) — built from reading the actual decision code, not guessed.
• Both modes, same test set, twice. Willows (PropFlow's own sandboxed test property, not a real customer) was flipped between "legacy" and "v2" mode and every scenario replayed through the real code path both ways, so the comparison is apples-to-apples on the same inputs.
• Three safety proofs before the real run: (1) nothing was written to the real production database — everything landed in the isolated dev/test database; (2) zero real messages went out — a live proof run showed the only "sends" were 3 internal calendar-sync notices to PropFlow's own operations inbox, never a real prospect; (3) zero paid AI usage — every judging/classifying call ran on the subscription plan, never a metered API key.

90 of the "incorrect" results on real Camellia conversations come from the same root cause: Willows (the sandbox we're required to test in, since we can't touch real Camellia data) doesn't have the same calendar availability that the real property had on those historical dates in 2026. So in both legacy and v2, the conversation gets stuck one step earlier — it can't even find an open tour slot to propose — before it ever reaches the confirm/cancel/decline decision we're actually trying to test. This happens almost identically in both versions (the two versions only disagree on 0.2% of these), so it tells us nothing about legacy vs v2 — it's a sandbox-vs-reality gap, not a decider defect.

Once those 90 are set aside, the more meaningful, adjusted correct-rate is:

Labeled "adjusted" deliberately — this is the honest, smaller, more meaningful sample, not the headline number.

The one genuinely different case, read by a judge: a real Camellia email conversation where a prospect emailed "I would like to schedule a tour" with no time given, then three weeks later wrote "I am here for my tour!" The real prospect had a genuinely scheduled 3:15pm tour that day, arranged through a channel this replay doesn't have a record of — but the test fixture that reconstructs this scenario has no starting tour time saved for it, so the code sees "prospect says they're here" with no tour on file to point to. Legacy's majority answer confirms a tour anyway, using a placeholder 2:00pm time that was never actually offered — which is itself a real defect (making up a time nobody agreed to). v2's majority answer says nothing at all — also a real defect (a prospect standing at the door gets silence). The judge's ruling: this is a shared, pre-existing gap in how either version handles "prospect claims to be present, no live tour proposal on file," triggered by a corpus test-data gap (the real tour time was never captured into the test fixture) — not a v2 regression. Both bad behaviors (silence, and inventing a time) showed up on both sides across the 6 reruns; the split just happened to land unevenly this one time.

Resolved as "zero is the real answer, not a measurement gap." Fede's original ask was specifically "report on how many it blocked, why it blocked, and if it was correct." The first run's guard-block and tool-call counts came back zero because of a field-naming bug — that got fixed, and the hardened re-run counted 1,575 recorded activity steps for legacy and 1,599 for v2 across all 1,289 turns per version, so the measuring equipment is now confirmed to be capturing real activity, not going quiet. But the specific "safety guard blocked an action" and "named tool was called" counters still read zero, and this time we traced why: this code doesn't have anything literally named a guard-block event, and the tour decider writes to the database directly rather than through a named-tool interface, in either version — so there's genuinely nothing for those two specific counters to count here. That's a fact about how this part of the code is built, not a leftover measurement hole. Everything else in this report (the state outcomes — correct/incorrect/needs-human, plus the reply text and step counts) is confirmed trustworthy.

Four independent checks, all pointing the same way:

• Every one of the 43 previously-known, real, customer-facing bugs — the actual incidents this system has caused since July — was replayed against both versions and produced the identical outcome on every single one. Not one case where legacy handled it correctly and v2 broke it. See the table below for the fixed/still-broken/needs-human breakdown per bug.
• All 484 real voice-call turns from Camellia's history showed zero disagreement between the two versions — the entire phone channel is a dead match.
• The one channel with real disagreement (text messages, 23% of turns) was rerun three times each to separate a real difference from the AI classifier being inconsistent with itself — nothing survived that check as a repeatable, direction-consistent v2-only failure. The two places it looked closest to a real difference (the RG-decline email fixture, and the "decline a confirmed tour" adversarial set) were checked directly against the current code and both fail identically in both versions — same shared bug, not a v2-specific one.
• A real live trial in production (below) ran actual conversations through v2 end to end and found zero cases where it produced a wrong or missing outcome that legacy wouldn't also have produced.

Real shared defects the adversarial set found (present in both legacy and v2 — not caused by v2, not fixed by v2)

• Declining a tour is unreliable in both versions. Depending on the exact wording, a decline sometimes does nothing at all (state stays unchanged, prospect's "no" gets silently dropped) and sometimes gets recorded as "cancelled" instead of a proper "declined" — there's no dedicated decline state that actually fires reliably in either version.
• Declining an already-confirmed tour is a silent no-op in both versions. We expected (from reading the code) that this might silently overwrite the confirmed tour to a wrong state, since the decline code path is missing a safety check that the confirm path has. What we actually saw in five separate test scenarios: the tour just stays confirmed and the prospect's decline is dropped on the floor — no notification, no state change, in both versions identically. Different failure mode than predicted, but still a real gap worth fixing.
• The email-decline case (from the regression test fixture) reproduces the same way in both versions — "that time doesn't work for me" on email leaves the tour in its prior "proposed" state instead of moving to declined, in legacy and v2 alike.

The one thing that looked odd — resolved, not a bug. The first email attempt used a sender address with the word "trial" in it and a generic template body. It never got a reply. Investigation found: the email did arrive, PropFlow's own webhook correctly received it — and then PropFlow's own AI spam filter correctly recognized "an address that says 'test' in it, with a generic templated message" as not-a-real-prospect and silently skipped it, exactly as that filter is supposed to work. The tour code (legacy or v2) never even got a chance to run — the filter caught it a step earlier. Rerunning with a normal-looking sender and a realistic message went through cleanly (row 5 above). Separately, this surfaced a real but unrelated gap worth its own follow-up: a genuinely real first-time prospect who gets wrongly caught by that spam filter today has no automatic alert — only a person manually checking a log would catch it. Not a v2 issue; filed as its own follow-up below.

• This is a one-sided replay. It shows the two versions matched each other on this test set — it does not prove either version is "correct" against some independent standard beyond what actually happened in the real conversation or what the scenario was built to expect.
• The test path skips four production safety checks that sit above the decider: Twilio signature validation, per-number rate limiting, duplicate-message dedup, and — most importantly — the code that intercepts "STOP"/"START"/"HELP" before it ever reaches the decision logic in real production. Any adversarial scenario involving opt-out keywords or duplicate delivery was testing the decider in isolation, not the full real-world path; scored and flagged accordingly, not blended into the headline numbers.
• The AI classifier that reads a prospect's reply is not perfectly consistent run to run. Rerunning every single-run disagreement three times each per version (the hardened re-run's 34, and the original run's 40) confirms this: the large majority settle as classifier noise where both versions land on the same answer once you rerun them, not a real mode difference.
• The classifier's own stated reasoning for each call still wasn't successfully captured in the hardened re-run either — an open observability gap, useful for future debugging, but it doesn't affect any correct/incorrect/regression finding in this report (those are all based on the actual tour state and reply text, both of which were captured and compared).
• The Willows sandbox calendar doesn't match Camellia's real historical availability for the replayed dates — this is why 90 of the real-conversation "incorrect" results are a replay-environment artifact rather than a decider finding (see "The numbers" above), and it's also the root cause of the one genuinely-different scenario in the hardened re-run (see "The hardened re-run" above).
• The live production trial's decline flow (row 3) didn't manage to test "decline an already-booked tour" specifically — real test conversations kept resolving into either an explicit cancel or a decline before a time was picked. Not a failure, just an honest scope gap in that one trial run.