Effort Raise — End-to-End Replay Validation

The shipped email/SMS drafting change (reasoning effort medium → high, reply ceiling 8K → 16K) replayed against 137 real production conversations through the full multi-turn pipeline with live tool calling — baseline vs shipped, plus the noise-floor control that makes the numbers interpretable.

2026-08-28 · replay bench: appfolio-45 test property (prod corpus read-only; The Willows is the voice sandbox — several replayed conversations are Willows leasing threads by content) · author: Claude (Fede's session) · all runs on the subscription, zero real sends (kill switch verified)

Verdict: the merged config is validated end-to-end. Across 324 aligned real-conversation turns, the behavioral difference between medium and high sits below the replay harness's own noise floor — two runs of the identical config diverge as much as medium vs high do (tool-call-set match 43.2% config-identical vs 48.1% across configs). Where divergences were graded, high was equal or better (it checks availability before booking a tour and tells the prospect the truth; medium booked blind). The one real cost is latency: +6.6s median per drafting turn. No revert warranted.

1 · What ran

The replay harness drove every conversation through the real production entry point — inbound dispatch, routing, the full agent tool loop, persistence — against the bench property on the dev table, reading the prod corpus read-only. Each arm replayed the same manifest: 137 real conversations + 16 synthetic suite = 153 conversations, 411 inbound turns, spanning leasing, tours, maintenance, renewals, vendors, and operational traffic on email and SMS.

Arm	Config	Conversations ok	Turns measured	Model calls	Tool calls	Turn latency med / p90
Baseline	medium · 8K (pre-merge main)	133/153	335	889	695	24.6s / 37.7s
Shipped	high · 16K (the merged PR)	136/153	360	1,005	920	31.3s / 59.5s

"Not ok" conversations are dominated by held turns — the escalation gate correctly holding inbounds for a human where the real conversation was in an escalated state. That is the pipeline working, reproduced faithfully. Zero messages reached any real person in any run (outbound kill switch armed and verified: 0 of 193 captured sends hit the wire).

2 · The comparison, and the control that makes it honest

Raw divergence numbers from a multi-turn replay are meaningless without a noise floor: two runs of the same code diverge through sampling, tool-loop ordering, and live bench timing — and each arm's own earlier replies feed its later turns, so small early differences compound. So the comparison comes with a config-identical control:

Comparison	Aligned turns	Exact reply match	Tool-call-set match	Judge: equivalent (of divergent)
medium vs high (the A/B)	324	18.5%	48.1%	43.9% (264 judged)
high vs high (identical config — noise floor)	271	16.6%	43.2%	— (judge off, deterministic metrics only)

Reading: the cross-config comparison diverges no more than the config-identical one — on tool-call sets it actually matches slightly better. The effort change's behavioral delta is below what this harness can distinguish from its own nondeterminism. That is the strongest available evidence of no regression: on 324 real turns, nothing systematic separates the arms.

3 · What the divergences actually are

Hand-graded sample of the sharpest tool-divergent pairs (full pair data retained locally):

High checks before it promises. Twice, the high arm fired an availability read before booking a tour and told the prospect plainly there were no one-bedrooms right now — offering to watch for one before their move-in date. Medium booked the tour without looking. Same class of judgment win the 42-scenario sweep saw.
One style nit at high: enumerating seven tour slots where medium offered two — the recitation instinct, already the subject of the separate voice-verbosity workstream.
Compounding drift, not defects: most "not equivalent" verdicts trace to earlier-turn differences feeding later turns (one arm books at turn 1, so turn 3 is a different conversation). Inherent to multi-turn replay; the noise floor accounts for it.
No factual errors, dropped commitments, or wrong escalations from high anywhere in the graded sample.

4 · Latency — the real cost, quantified on real traffic

Metric	medium	high	delta
Turn latency, median	24.6s	31.3s	+6.6s (+27%)
Turn latency, p90	37.7s	59.5s	+21.8s
First drafting call, median (email)	4.9s	9.3s	+4.4s

Email/SMS are asynchronous channels; the decision doc accepted seconds-scale thinking cost for judgment quality. Voice is structurally unaffected (different engine, no effort parameter). First-24h Langfuse watch stays on: latency, truncation stop-reasons (none expected at 16K — zero replies approached even 8K), cache cold-starts.

5 · Coverage ledger for the whole decision

Authored suite (37 scenarios, judge-pinned): medium 32/37 · high 34/37 · xhigh 33/37.
Real single-turn replays (28 rows × 3 reps, Fable-graded): 77/84 tie; medium's unique miss was a judgment error, high's was stylistic; xhigh added 13–27s tails.
Lane coverage pass 2 (42 rows: maintenance, turnover, vendor, day-of-tour, capability, spam × 3 efforts × 3 reps): high 42/42 · medium 40/42 (leaked internal narration into two resident-facing replies) · xhigh 41/42 (recited internal emergency taxonomy at a worried parent).
End-to-end multi-turn replay with tools (this report): delta below noise floor; graded divergences favor high; latency +6.6s median.
Post-deploy live check: real-path SMS through prod, trace-verified (session 004).
Not covered, stated honestly: collections and tenant-side billing have zero real traffic to replay and were not synthesized; voice is out of scope by architecture.

6 · Methodology findings (worth keeping)

The shared-salt two-arm workflow grades state as behavior. Running arm 2 with arm 1's replay salt makes it meet the bench as the same synthetic people — so it truthfully says "you already have a tour Saturday at 11" because arm 1 booked it, and the judge scores that as divergence. Measured: that protocol reported 33.9% equivalence vs 43.9% fresh-vs-fresh. Recommendation (small harness PR): document fresh-salt arms + a config-identical noise-floor run as the standard protocol; a --allow-salt-mismatch flag for parity is drafted in the working tree.
The bench billing guard works. The first baseline attempt failed 153/153 with the bench lane's fail-closed refusal to bill the metered key — exactly its job. The harness's sandbox preflight checks a credential exists but not that it works; a 1-token probe in the preflight would turn an hour of dead run into an instant error.
Real quota is finite: by evening the pool returned genuine rate-limit 429s (with real reset headers — distinct from the fake client-gate 429s documented in credentials lore). The A-vs-A noise floor was therefore computed judge-free from the two existing high runs rather than burning a fourth full run.

7 · Follow-ups queued

Harness PR: salt protocol documentation + parity flag + credential preflight probe.
Drift test pinning the A/B script's copied constants to the product constants (bot advisory, agreed).
First-24h Langfuse watch on the live config.
Voice conciseness ship path (separate report: voice-verbosity-report-2026-08-28).

8 · Provenance

Runs (result dirs, local): baseline-medium 15:23Z (133 ok/335 turns), candidate-high 16:04Z (136 ok/360 turns), candidate-high-salted 17:02Z (contaminated-protocol arm, kept as methodology evidence), baseline-medium-2 18:12Z (invalidated: main had already moved to the merged config, and the run hit real 429s). Parity reports: fresh-vs-fresh with judge (264/264 divergent turns judged), config-identical judge-free. Corpus: manifest built 2026-08-28 from propflow-prod, read-only, synthetic replay identities only (no real PII leaves prod). Billing: subscription bearer on every model call (verified in each run's sandbox proof). Decision doc for the original raise: effort-sweep-email-sms-2026-08-27. PR: #6461, merged 2026-08-28 15:26Z.