Vendor calls, before & after

Every real vendor call we have on record, replayed against the redesigned Clara · 66 of 69 calls · shipped — measured on the state that merged (ddce6771, PR #4805, merged 2026-07-29) · run 2026-07-28

The headline

Where Clara recognizes the vendor, the redesign works. On the 27 calls where she knows who is calling and what job they are on, she now resolves 20 of them herself. In production those same calls went to a human almost every time. She reads the vendor their purchase-order number — 618, 629, 642, 661, 667, 670 — on calls that previously ended with a person hunting for it on a paper list.

She never made a number up. Across all 66 calls there is not one invented purchase-order number, not one internal record ID spoken aloud, and not one case of reading a list of numbers at a vendor. That was the thing most at risk from giving her the ledger, and it held.

The three cohorts

The corpus itself splits these calls: 27 are ones where knowing the purchase order changes the outcome, and the rest are cold calls, sales pitches, wrong numbers and jobs we have no record of. The redesign only claims to help the first group, and the numbers follow that line exactly.

The thing this page asked for last time — fixed

The previous version of this page led with one finding: of the recognized calls that still reached a human, 8 of 10 opened with the vendor asking for a person by name — "Front desk." "Speak to Joanna." — and Clara transferred on the spot while holding the exact number they were calling about. The flagship example was three turns long, with PO 682 in her hand.

That call now goes the other way:

Person-ask transfers among recognized vendors dropped from 8 to 5 — that figure belongs to the earlier 213c1937 candidate; the shipped run does not carry a person-ask classifier, so it is not re-measured here. PO 682 and PO 644 are now given to the vendor on the call instead of by a human afterwards. A third — the unit 615 blinds order, PO 646 — was resolved on the call in the previous build but transfers again in this one; that is the inconsistency described below.

What the earlier candidate broke — and now doesn't

An earlier build of this change introduced four problems, which this page reported. All four were fixed before shipping, and this run confirms it:

One thing was worth checking on its own: an early attempt at the person-ask fix went too far and made Clara refuse to put people through. That would be a worse failure than the one it replaced. It is not present here — on every call where the vendor asked again after Clara's offer, she put them through. She offers once, then yields.

What is still left

The offer is not yet consistent. On some calls Clara transfers on the first turn without first mentioning what she is holding — including one call where she has the purchase order for the job in front of her. Yielding is right; yielding before offering is the remaining gap, and it is tracked as a follow-up.

Honest caveats

Every call, side by side

Recognized vendor calls first — these are the ones the redesign is meant to change. "What really happened" is from the labelled record of the production call.

How the mechanical checks did

Alongside the verdicts, every call is checked for things that must never happen regardless of how good the conversation sounded. Across 66 calls there were 4 failures, and none of them were an invented number:

• Three calls where she promised to pass something along twice — meaning the first promise did nothing.
• One call where Clara asked two questions in one turn instead of one.

No invented purchase-order numbers. No internal record IDs read aloud. No reading a list of purchase orders at a vendor. No telling an established vendor they might not be in the system.

Where this came from

The replay harness, the per-call scripts and the full transcripts of all 66 replayed calls are in PR #4785 (evals/gauntlet/run-vendor-corpus.ts, with the raw run saved under evals/snapshots/). Every figure on this page is generated directly from that saved run, with the one exception flagged above (the 8→5 person-ask count, which comes from the earlier candidate).

Corrected 2026-07-29 after a source audit.