Testing playbook: regressions and new features

1. The ladder

The Willows bench is shared. Two other rigs wipe every open work order on the Willows (appfolio-45) nightly: Agent Smith’s maintenance eval about 07:00–07:45 UTC and the CI “Clara Behavior Evals” run about 03:40–04:30 UTC; Gera’s fast-lane scripts also dial the same line. A live run there never crosses those windows, claims the bench on #engineering-bridge before each cycle (a BRIDGE CLAIM with the window), and grades a work order that vanished between a read and a re-read as interference, never as a product defect.

2. Fix vs new feature

Why the split: three improvements to live behavior once shipped behind settings that nobody pressed, and a sync guard sat off at three customers. A fix that waits for a switch is a fix nobody gets.

3. The two customer shapes

4. Test companies: what exists, what we keep

Why no company is ready: the daily renewal check scans the whole platform, so turning a building on also scans every old lease already on it; the recipient guard lets staff email through and fails open; held messages fall back to #alerts when a company has no channel; and nothing copies settings between companies or cleans a multi-building seed. Those four gaps are the loader.

The standard (Fede, 10/9: "fairhaven and willows thats it"): Fairhaven for centralized leasing over distributed properties, Willows for a single community. Two test companies, never a third. Fairhaven can carry Western Slope's and Situs's anonymized books side by side, each under its own prefix, when a run needs more realism. Each is in sandbox PMS mode, Supervised, with a test Slack channel, contacts pointed only at registered team phones and inboxes, and is rebuilt by one command from an anonymized copy of a customer's book with a building map. "Clean up" becomes "delete and rebuild", which is also the proof that the loader works.

Decisions

• E9. Decided 10/9 (Fede): Fairhaven is the practice company for Western Slope's shape; both mirrors are deleted once the guest-card replay harness targets Fairhaven. The loader targets Fairhaven (its 11 buildings take the 27 fixture buildings via the building map; more buildings are added with the existing provision script if a run needs them).
• E10. Done 10/9: Test Client Co and Module Gate Probe deleted, re-query shows zero rows. The delete handler left 9 membership rows for shared test logins, cleaned by hand; gap filed as #18105.
• E9 done 10/9: both mirrors deleted; the guest-card replay harness now seats both real companies' homes in Fairhaven. Side effect to know: because Fairhaven is shared, the harness no longer copies a company's company-wide settings and knowledge onto it (per-home facts still copy). If a replay needs company-wide knowledge, that is a follow-up (#18113 has the nits).
• E11 leftover (pending Fede): Fairhaven still holds 185 rows copied from the Willows AppFolio property (113 guest cards, 31 applications, 41 tenants) by the sync before its six buildings went to sandbox mode. JP&Co's own company has none of them. Delete? yes / hold.
• E11. Done 10/9 (Fede: clean): Fairhaven cleaned with the existing kit (4,737 rows, 203 batches, nothing refused), jpco-bound buildings on sandbox PMS mode. Leftover: the six emptied buildings go to sandbox mode too, and 185 stray JP&Co rows get cleared; the deletion backup (personal data) is removed by 10/11.

Overnight: the loader (prepare, seed, cleanup, dry run) merges as scripts; its first real target is Fairhaven once E11 finishes.

5. Pass bar and the report

The report is always before → after: what the tenant or the team experienced, what they experience now, and the thing that proves it (a delivered text, a scorecard, a diff). "Much better, and here is how you can tell."

6. Rules that keep this honest

• A check with no mutation control does not count. Two "engine defects" in one night were grader errors a control exposed.
• Real production data before design conclusions: pull the records, read-only, then decide.
• Test runs contact only registered team numbers and inboxes. The bench line is not a person.
• No harness or prototype enters CI; it runs locally and ships its result as a page. It enters the repo as one small PR after the decision.
• Camellia is the regression oracle for renewals: it stays byte-identical through every slice or the slice does not merge.
• Turning anything on at a customer is Fede's call. Everything below that is ours.
• A list or report that names real people is checked row by row against the records before its first customer send (added 2026-10-16). "Byte-identical to yesterday" proves the code did not change; it does not prove the rows are right. The night before the first leasing call list went out, a per-row check against the raw records found a lead who had texted STOP, a no-show recorded only in a staff note, and listing-site inquiry forms counted as the lead replying; four earlier dry runs had looked fine. Fede: "I don't understand how we are discovering all these issues so late." The check: every row, group and quoted line traced to a record id; a sample of the excluded rows traced to the reason they were excluded; findings fixed before the send.

Per-release model and refactor decisions (new model out, big layer rebuilt) use a separate procedure with its own frozen set, budget and verdict: Clara Bench (proposed 2026-10-23). The ladder on this page stays the per-change proof; Clara Bench is the per-release decision. Its standing rule for every run: agree the bar and the budget before the run starts.

7. What "tested" means from now on

Proposed — pending Fede's review (2026-10-17)

Why now. The staff leasing desk (the text line a leasing agent uses to tell Clara about no-shows, notes and tours) failed a trivial ask at a customer after 59 commits in three weeks, unit tests on every one, and a 22% replay on Sept 30 that nobody acted on. One bench run of simulated agents then failed 21 of 45 scenarios on main in two hours, including the customer's exact case.

The four proofs

The attack fleet: a standing capability

• Cell: action × persona × data shape. The staff desk has up to 2,688 (14 × 12 × 16); the first run covered 45.
• Scenario: a seeded production data shape, a scripted first text, a persona that writes the rest, and the expected outcome in plain words. Graded only from rows and four deterministic reply checks: a promise with no record behind it, an internal term named, a question already answered, an instruction that cannot work. A tone judge may comment, never decides.
• Isolation: bench companies only (Willows, Fairhaven). Names, emails and card tags salted per run, Twilio magic-number phones, outbound suppressed (the run stops if not), teardown by tag, zero rows after. One process per bench; the salt covers people and phones, not the property row.
• Result file: one line per scenario, rendered to a report: ranked findings (what staff said, what Clara did, what should have happened, how often a real user hits it), scoreboards by action, persona and data shape, stray sends, teardown check.
• Catalogue: ranked defect classes, each verified or refuted by the two skeptics (a busy calendar is a bench artifact), with a severity and a fixture-worthy list.
• Budget: Fede authorized dozens of agents and heavy token use. Subscription only.
• Open gap: one bench teammate Clara notifies still has a real number (suppression held); swap it before the first nightly.

The 22% rule

• A number nobody acts on is a defect of the process. Below the bar: a verified failure with no owner, or a grader under its baseline.
• Baselines are recorded before a redesign (staff desk, 10-16: 24 of 45 pass). The redesign must beat them on every grader, not on average: each action, persona, data shape and reply check.

Where it sits, who runs it, when

• Rung 2 (new), adversarial fleet on the bench: between unit tests with a planted defect and the real-book replay. Required for any change to a lane that talks to humans (staff, prospects, tenants).
• Fixture pack: every production incident becomes a fixture within 24 hours, before its fix merges, red run captured at the pre-fix commit. Scrubbed, in the shared harness, each check with a mutation control.
• Every PR in a human-facing lane names the rungs it climbed and links the results file.
• Nightly on main for the staff desk, prospect lane and tenant lane: on the subscription (not CI), outside the Willows wipe windows, bench claimed first, results on the initiative's page. A red nightly pages the owner lane, never Fede. staff desk exists prospect, tenant fleets to build

What it is not

Not a kill switch (fixes still ship with no switch), not a human approval queue, not a reason to hold a PR (a missing proof is work for the lane). It is how we prove a change before merge and find the next incident before a customer does.

8. Live test at the Willows: the one recipe

added 2026-10-22, from the first document-engine run (renewal packet parked in 7 s, then cancelled) · Fede: "the Willows should never ask for permission to change things"

The Willows (test property 45 in JP&Co's AppFolio, PropFlow property appfolio-45, company org_sandbox) is the only place a session may change things. Writes scoped to it are pre-approved; a write that touches a shared JP&Co object (a template or setting other properties use) still asks. Western Slope, Situs and Camellia are read-only, always.

The run, in order

1. Preconditions, reads only: the merge is live on web and on the worker (health commit, worker build id); the Willows unit shows the expected starting state (robot screenshot plus a robot read); the dry run of every write prints exactly what it would change.
2. Start, one shell command: set the Willows rung, sign in, call the route, restore the rung. If the route refuses, restore anyway; nothing was minted.
3. Watch, one blocking loop: the lease record every 90 s until the flow records its outcome (parked / blocked / refused), plus the Temporal history and the worker log for the lease id. Never poll from the conversation.
4. Evidence: robot screenshots of the unit and the document, the robot's answer with the ids it created, the worker log lines, the lease record. Look at every PNG.
5. Cleanup, in this order: cancel the AppFolio write through the same robot verb with an expect on the ids it created → stop the workflow → reset the one tenant's PropFlow state (always with the tenant id; the bare route resets every open saga) → release the claim rows the start created → take the test recipe or setting out of use → confirm the rung is back. The lease keeps its outcome record by design.
6. Report before → after on the initiative's hub page with the screenshots, and the tracker row gets the proof link. The run plan with the exact commands lives next to the proof.

Closed 2026-10-23: no prod staff login is needed for Willows runs. The first two runs on Oct 22 improvised one (the MFA e2e account, un-waitlisted per run); that was the mistake. Start and reset flows directly, as above.