Turnover projection rethink — before / after
The projection used to ask one model call to do two jobs: read the walk, and decide the money. It now does one LLM extraction with verbatim quotes and no money fields, decides every money question in pure TypeScript, and escalates only what the rules cannot settle to a judge with a 3-draw unanimity panel behind it.
Headline
| BEFORE single-pass v14, opus-5 | AFTER extract → decide → judge | |
|---|---|---|
| charge_correctness (52 scenarios, LLM judge, median of 3) | 78 | 98 |
| per-scenario mean | 75.3 | 92.3 |
| scenarios scoring 0 | 9 | 3 |
| scenarios scoring 100 | 33 | 44 |
| vendor dispatch billed to the tenant ("call Cardenas for trash removal") | 2 of 3 draws | 0 of 3 draws |
| that scenario's score | 0 | 90 |
| money decisions made by pure code | 0 of 181 | 171 of 181 (94.5%) |
| decisions left unresolved for the PM | — | 0 |
| scenarios needing any judge call | — | 4 of 52 (24 calls) |
| fabricated quotes caught by code | 0 (no check existed) | 2 |
| hard failures | 0 / 52 | 0 / 52 |
| projection latency, mean / median | 9.5 s / 9.2 s | 29.9 s / 23.7 s |
15 improved · 33 unchanged · 4 regressed. The ladder is 3.1× slower per projection at the mean, 2.6× at the median — verbatim quotes cost output tokens and strict decoding is not free, and both are what the fabrication check validates against. Flagged, not optimized away.
The four regressions, named
| score | what it is |
|---|---|
| 100 → 0 | Locked policy working. A walk documents a wrecked carpet with no charge instruction anywhere. BEFORE attributed it to the tenant on damage language alone; AFTER refuses. The grader rewards inventing the charge; the rule forbids it. |
| 100 → 50 | A real remaining defect. The proposer's item axis never checks attribution, so a standalone damage item still mints a line at the KB typical with nobody behind it — the same defect class the scope axis no longer has. Outside this PR. |
| 100 → 83 | Missing category mapping. A correctly attributed, correctly priced $250 pest-control charge files under damage because the taskType→category table has no pest_control row. Wrong label, right money. |
| 92 → 88 | Grader noise. One partial verdict on an adversarial transcript; both arms mint the same charge and the graders disagree with each other more than the arms do. |
The first row exposed a gap that #5002 did not fix: an unattributable finding produced a silent zero — no charge line and no question. That follow-up has since shipped (#5034, merged): an unattributable finding now mints a pending who-pays question that no bare "yes" can turn into a deduction. Measured across the same 52 walks: 65 questions total, median 1 per walk, p90 2, max 4.
What the replay corpus caught that unit tests did not
Five money bugs found by scoring an arm, none by a test. Each is fixed and pinned, and the AFTER arm was re-run from scratch rather than reporting a number for code that isn't shipping.
- Case-sensitive dispatch verb — a vendor call-out billed the tenant $150.
"three fifty"parsed to $53, returned confidently instead of escalating.- Bare-amount shape dropped — "charge the tenant 150" produced no amount at all.
"<work> for <N>"dropped — five real amounts lost on a single walk.- Uncovered-item rescue minted a
damageline at the KB typical with no attribution behind it.
Determinism
Neither arm is byte-stable — 0 of 6 probe cells — and no Claude 5 model can restore the temperature-0 contract, so byte-stability was the wrong target. What changed is where the variance lives. Across three draws on the dispatch scenario, BEFORE went 0, 2, 2 tenant tasks: two charges appearing from nothing but a resample. AFTER structurally cannot add one — the decide stage is pure and byte-identical by construction, pinned by a repeated-call equality test. The remaining variance is extraction recall and wording, not money.
Status
- Implementation PR #5002 — merged, live in production (Fede accepted the ladder 2026-07-30); legacy single-pass path deleted in PR #5060 (superseding the auto-closed #5007); decision recorded as ADR-0121 (Accepted)
- Follow-up PR #5034 (unattributable findings ask who pays; flooring files under Carpet) — merged
- Decision record: ADR — turnover money decisions (proposed)