Real-caller replay — baseline scorecard

Camellia · run 2026-08-14 · shipped with the replay-corpus grader (PR #5765, merged after a 4-round review)

In plain terms

We took every real phone call Camellia received, pulled out the genuine questions callers asked, and asked each one again to today's Clara — grading her against today's actual property data. First reading: she passed 12 of 19 and "invents answers." Then every failure was adversarially re-verified, and 5 of the 7 collapsed — the judge had punished behavior Clara's instructions explicitly script, because the judge can see the property data but not the instructions. What actually survives: one real fabrication (claiming same-day availability that didn't exist) and one low-severity phone-delivery slip.

The result

real questions, from 71 calls

handled right (8 answered, 4 correctly handed off)

real misses after adversarial re-verification (5 of the run's 7 "wrong" verdicts were grader error)

The 7 misses, after adversarial re-verification

The caller asked	Final verdict
¿Tienen citas disponibles hoy? (any appointments today?)	real fabrication Claimed same-day availability when the slot list held nothing for that day. The one clean invention in the run.
What are the fees?	real, low severity Quoted one labelled deposit tier before asking unit size — her phone-delivery rules say ask first. (Volunteering the special was scripted, not a violation.)
Can we get a virtual tour?	cleared Her instructions state "we support virtual tours at every property" and script exactly her reply; the offered days matched real slots. The judge couldn't see the instructions.
Who gives the tour?	cleared The property data the judge was shown names Erika as the on-site manager to ask for, and Clara's rules order her to state the guide as fact.
Is that 9 AM Central Time?	cleared The Denver address was on file; production even hands her "MDT" in its date string, which this rig had stripped.
What's available and how do we move in?	cleared "Apply online and we book from there" is her scripted model answer to this exact question.
What time is my appointment today?	cleared "No tour on file → say so plainly and offer to book one" is a hard rule; the grading criterion demanded the opposite.

What this actually proves

Clara's data-grounded answers were solid across the board, and most of what first read as "inventing answers" was the grader being blind to her instructions: it sees the property data but not the prompt rules, so every rule-scripted behavior (virtual tours, naming the tour guide, offering to book) graded as fabrication. Two fixes queued for the measuring stick itself: show the graders the prompt-level policy constants, and pass the real production date string (with its timezone) instead of a bare date.

Next step

The one real prompt fix: Clara must never assert a day or time is available unless it's in her slot list — attempt the booking and let the result speak. Tracked: the invented-answers card, re-scoped to this.
Rig fixes (grader sees prompt constants; real date string) tracked on the harness-validity card, then re-run for an honest baseline.

Source: scripts/voice-harness/replay-corpus/ in the app repo · full report with verbatim replies: reports/replay-grade-2026-08-14T23-24-48-792Z.md (kept out of git by design — caller-derived text). Answer + judge model: claude-sonnet-4-6, temperature 0.