Turnover Before/After: replaying every real scenario through today’s code

Method

• BEFORE = what production actually held for that capture: the condition-report notes plus the projection prod stored, with its promptVersion provenance — read read-only from propflow-prod. Where prod has since deleted the row, the 2026-06-10 ground-truth snapshot committed in the walk-call eval corpus stands in.
• AFTER = today’s real handlers, no mocks: handleStartInspection → handleAppendNotes×N → projectConditionReport (live LLM) → assembleTurnoverCharges (real propose + judge), in a JSON sandbox with the DynamoDB table name deleted at module load.
• Identical inputs where possible. For the 27 harvested walk calls the replay feeds the append_notes parameters the live voice agent actually fired, verbatim and in order — not a re-segmentation of the transcript. For the 25 report-anchored captures it feeds the stored notes back through the append pipeline block by block.
• LLM on the Claude subscription throughout, never the metered key.

Results

The cleanly-comparable subset (n=20)

The charge row is not like-for-like and shouldn’t be read as one: the charge layer did not exist until 2026-06-08/09, and production never persisted a single charge row for any of these 46 turnovers. Every dictated dollar figure — $150 clean, $250 clean, $300 carpet, the $200 cleaning edit — died in the conversation. That is a capability added, not a defect fixed.

Quality scorecard (five dimensions, LLM judge, verbatim transcript evidence)

Finding 1 — the projection’s reproducibility guarantee is void

condition-projection.ts pins CONDITION_PROMPT_VERSION with an explicit instruction to bump it whenever the model changes. A PR on 2026-07-27 changed the default agent model to claude-opus-5 and did not bump it. Independently, claude-opus-5 rejects temperature with a 400, so the projection call now omits it — the “temperature 0, reproducible” guarantee that ADR-0034 and every projection eval rest on no longer holds.

Measured, not inferred.

Same version string, two different models, two different sampling regimes — provenance can no longer tell them apart, so no stored report can be attributed to the code that produced it. Three repeat runs of three scenarios: item counts held steady (8/8/8, 9/9/9, 7/7/7) but item labels drifted run to run (Paint ↔ Paint / walls, Wall (anchor wall) ↔ Wall). Structurally stable, textually not.

Finding 2 — the duplicate condition-report race is still open

Production holds five groups of duplicate move-out reports: 16 rows for 5 real captures, so 11 spurious reports. One customer turnover has four reports stamped the same minute; one sandbox turnover has five, then three more.

There were two candidate mechanisms. The cross-call capturedBy fork was closed in June. The concurrent/retried start_inspection race was not. Every duplicate group shares one identical capturedBy — so it is the race, and the race is still reachable today:

The voice provider makes this easy to hit: it cancels a tool client-side with “Tool execution was abandoned due to user input” while the server call completes, and the agent then retries. That exact sequence is visible in one harvested call at t=25s and t=34s.

Latent hazard on top: getConditionReportByTurnover returns items[0] under a comment asserting the partition holds at most one row. Production disproves the comment, so that reader returns an arbitrary report — and the dangling-pointer recovery path added in June depends on it.

Finding 3 — the confirm-leg stress harness is green and blind

mintLocalWorkOrderId uses the DynamoDB document client and table name directly — an atomic counter that does not honour DATA_BACKEND=json. In any JSON sandbox it throws ResourceNotFoundException, scopeTurnover rethrows it as a non-retryable failure, confirmTurnoverScope catches it, and the turnover ends with zero tasks and zero work orders.

Verified twice. Replaying the two real historical PM approvals plus three more, all five returned:

… advanced the stage to work_scheduled, stamped estimateApprovedAt, and persisted 0 tasks and 0 work orders. Then, as a control, the existing confirm-stress.ts harness run unmodified on main: 16/16 PASS with the same failure firing inside every scenario. It passes because its checkScoped asserts stage + estimateApprovedAt + one tasks_scoped event + a note — and never that a task or a work order exists. The confirm-leg suite has never exercised work-order minting.

Two separable facts, and it matters which is which:

• The fail-open is deliberate and documented — the stage advance has already persisted, the reply path must never throw, so it logs loud and a recovery sweep is expected to catch it. It is still exactly the “Approved did nothing” shape the PM experiences, and production says the trigger is not hypothetical: 41 of 50 child work orders on these turnovers are sync_failed.
• The harness blindness is a bug, and it is why nobody noticed.

The scope decision layer is correct: it computed 2 tasks for the customer unit and 2 for the sandbox unit, matching history exactly. It is the mint step that cannot run in a sandbox.

Finding 4 — a money-corrupting mis-attribution that production actually shipped is fixed

The clearest single before/after in the corpus. A sandbox walk on 2026-06-10. Production’s notes ended with Charge tenant two hundred fifty dollars. and Covered by property, not charged to tenant. sitting as orphaned lines under the “Full cleaning needed” block — the $250 belonged to pest control and the property-covered attribution belonged to the window. The sentence-level dedupe of that era had eaten both lines from their real findings and left them attached to the wrong one, which is how a cleaning line ends up carrying someone else’s $250.

Replaying the same three append_notes payloads through today’s block-level dedupe — which landed three hours before that very call — produces the correct notes: both amounts bound to their own findings, nothing orphaned. Same input, corrupted then, right now.

Finding 5 — the customer property barely exercised the pipeline at all

This is the honest answer to “do we have the whole context”: mostly we do not, because there is very little real customer history to have.

• 26 customer turnovers. Two ever got past created + inspection_scheduled into a walk with findings, and exactly one completed the loop — PM approved, 2 work orders created and synced to the PMS. Across both properties, 31 of 46 turnovers never got past two timeline events: auto-created by the notice-to-vacate sync, then nothing.
• All six harvested customer-property walk calls were dictated from the internal test line — i.e. internal tests against real customer units. Three join to production conversation rows that confirm the number; the other three have no conversation row at all, and their siblings minutes either side are all that line. The walk-corpus README claims one of them was dictated by the real PM; that could not be confirmed, and the phone evidence points the other way.
• Zero vendor call sessions on the customer property. All 56 are sandbox. 50 belong to a single turnover with 45 child work orders — a scripted outcome-permutation sweep, not a turnover.
• Exactly one conversation in the whole turnover-related set is a real resident on a real customer unit.
• No charge row type exists. Every turnover partition holds only timeline events. estimateApprovedAt is set on 2 of 46 turnovers. estimatedCost is null on all 50 work orders and category is general on all 50 — no trade differentiation was ever persisted.

So the pipeline’s real history is: one complete customer loop, one complete sandbox loop, roughly forty internal test walks, and a large amount of sync noise. The autonomy-confidence question is answered here for the walk-capture and charge-assembly lanes. It is not answered for the approval lane at customer scale, because that record does not exist yet.

Finding 6 — the one possible regression in the whole exercise

A separate like-for-like A/B: the 6-entry corpus from the 2026-07-22 harness run, re-driven through today’s code and re-scored with the same grader.

Capture and grouping improved reproducibly — that is the filler filter landing. The charge_correctness drop is not a clean regression and should not be reported as one: today’s code produces charge packages on 4 of the 6 calls where it produced 2 before, so the denominator changed and the newly-scored calls score low.

There is exactly one like-for-like case, and it reproduced in both re-runs. Verbatim dictation:

Today that becomes Damage — "Trash removal from unit …" — amount: null (needs_amount) — attribution: tenant ("trash left behind by the tenant at move-out"). A vendor dispatch instruction was converted into a tenant-attributed charge with no amount. The judge failed it, and the judge looks right: “call Cardenas” is an instruction to place work, not a statement that the tenant owes for it.

Labelled honestly: possible regression, n=1 call, reproduced twice, worth a targeted regression test — not a headline. It is the only candidate the entire 52-scenario corpus produced.

Eras the replay confirms are gone

Coverage — what fraction of history was replayable, and why the rest was not

52 of 107 scenarios (49%). The 55 that were not break down cleanly, and nothing was skipped for convenience:

• 38 — no capture artifact: the turnover never had a walk or a report. Not a harness gap; there is nothing to replay.
• 14 — a walk call where no append_notes ever fired and nothing was captured (mostly the 2026-05-26/27 webhook outage, plus short calls that hung up before dictating).
• 3 — a report with empty notes.

Within the 52 that ran, only 20 give a clean verdict. All 9 customer-property replays are partial-input — every one of its captures either carried photos or shared a report with another call — so the customer property contributes zero clean verdicts. That is the single biggest hole in this exercise, and it is fixable: harvesting the photo bytes into the corpus (they sit in the photo bucket and are referenced by key on every report) would convert 27 partial-input scenarios into full comparisons.

25 of the 52 needed a synthesized outgoing-tenant occupancy and lease because production’s rows were deleted — handleStartInspection fails closed without a lease whose person matches the outgoing tenant. The shell carries no findings and no money, only the ids the lease-anchor guard needs. The utterances in those replays are entirely real; only the container is rebuilt.

Lanes not replayed — marked pending, not faked

• Multi-turn PM SMS approval/edit dialogue. The historical rigs exist: a 230-message thread with 78 confirm_turnover and 36 apply_charge_decision calls, human turns alternating “looks good, go ahead” / “change the cleaning to $200”; a 98-message dispatch rig; an end-to-end proof harness. Replaying them needs the multi-turn orchestrator that had not landed. What could be replayed of this lane — the single-turn confirm_turnover on the two real approvals — is Finding 3, and it is blocked at the mint step.
• Vendor dial-out. 56 sessions with inline transcripts exist; no dial was placed and none was replayed. One line found while reading them is worth a follow-up on its own: the agent told a vendor “The PO’s been issued, so you’re good to go — that’s the authorization to proceed” on an unapproved job.
• Photo/vision capture, per above.
• 12 walk-recap voice dictations that fired no turnover tools at all — real dictation that persisted only as timeline notes. Whether today’s agent would fire the tools is an agent-level question a handler replay cannot answer.

What to do next, in order

1. Bump the projection prompt version and record the model + sampling regime in provenance so a stored report can be attributed to the code that made it. Decide explicitly whether losing temperature 0 is acceptable for a money-bearing classifier.
2. Give start_inspection an idempotency key and a conditional write on turnoverId; fix getConditionReportByTurnover to stop returning an arbitrary row under a false comment.
3. Make mintLocalWorkOrderId respect the data-layer backend, then add a work-order-exists assertion to checkScoped. Until both land, no sandbox test can prove an approval creates work.
4. Attack charge_correctness (53) and vendor_roster_match (67) — 34 and 3 evidence-backed items respectively are already written up with verbatim quotes.
5. Harvest photo bytes into the corpus to convert 27 partial comparisons into real ones, which is what would let the customer property contribute a verdict at all.