Overnight tests — the full list
Read from the live sources on 2026-11-01: the mini's scheduled jobs, Smith's 02:00 nightly queue, and GitHub's scheduled workflows. All times US Central (the mini's clock). "Overnight" = 20:00 to 08:00. Last result = last night's run, or the last weekly run.
Count: 20 overnight tests. 15 test how the product behaves (9 every night, 6 weekly), and 5 are code-only checks that never touch the live system. Machine housekeeping and watchers are listed separately at the bottom and not counted.
Nothing has been removed yet. The retire-overnight-tests session has only set up its tracking board (two open PRs, neither merged).
What matters for the decision
- Three tests have been red for weeks with no fix: maintenance eval (89+ nights), spine health (49) and Morpheus (41). Right now they cost money and attention and don't change anything.
- The ones real calls don't cover: the opt-out block, live voice settings matching the code, renewals (harness and evals), turnovers, the AppFolio write-and-read-back, and the five code-only checks. Removing these removes the only check we have on them.
- The SMS and email stress tests against production are already off overnight (on demand only, since an earlier change).
- Cost: only two tests have a documented cost (Clara behavior evals about $8 a night, renewal evals about $2 a week). The rest aren't measured per test; the spend digest only tracks totals.
1. Tests of how the product behaves (15): full table
| Test | When | What it checks | Touches production / real data? | Last result | Cost | Real calls already cover vs only this catches |
|---|---|---|---|---|---|---|
| Spine health | Nightly 02:00 | Scans all production records for broken links: duplicate people, a lead and their conversation pointing at two different people, orphaned records. | Reads production, all customers. No writes. | Red 49 nights running (1 doubled tenant, 6 leads split into two people). A real data finding; a fix is in progress. | No AI. Free. | Real calls can't see this. But an always-on production detector already reports the same finding every night, so this mostly duplicates it. |
| AppFolio sync canary | Nightly 02:00 | Changes a phone number on a dedicated test tenant and test vendor in AppFolio, reads it back, then puts it back. | Writes to the test building in AppFolio only. | Pass | No AI. Free. | likely only coverage Likely the only daily proof that writing to AppFolio and reading it back works. Real work orders also write to AppFolio, so a break would surface within a day anyway, but in front of a customer. |
| Maintenance eval | Nightly 02:00 (runs 40 to 66 min) | 21 scripted maintenance conversations (texts plus some voice) against the test building. Checks Clara files the right work order, asks for photos, relays to the handyman, and so on. | Creates and cleans up real work orders on the test building only. No texts to real people. | 18/21. The same 3 handyman scenarios have been red 89+ nights. | Uses Clara's AI for every scenario. Not measured per run. | Real calls cover the common maintenance paths heavily. Only this exercises the handyman relay and close-with-photo scenarios on purpose, but those have been red for 89 nights with nobody acting on it. |
| Turnover eval | Nightly 02:00 | 2 move-out turnover scenarios. Checks the right trades (painting, flooring and so on) become work orders. | Creates 6 real AppFolio work orders on the test building, then reverses them. | 2/2 pass | Uses AI. Not measured. | likely only coverage Turnovers are low volume in real life, so real traffic tests this rarely. |
| Morpheus nightly | Nightly 02:00 (after the above) | Re-runs 3 approved generated maintenance scenarios to catch Clara getting worse. | Test building only. | 1/3. Red 41 nights; the cause has changed since Oct 7 and isn't pinned down. | Uses Clara's AI per scenario. Not measured. | Overlaps the maintenance eval and real maintenance calls. Red for weeks with no owner. |
| Clara behavior evals | Nightly 21:47 | The full scripted Clara eval sweep across every area (maintenance, leasing, renewals, routing and so on), split into 4 parts. | No. Mocked data. | Failed (3 of 4 parts red) | About $8 per sweep. The workflow notes 6 sweeps cost about $50. | The same suite already runs on every PR that changes Clara. The nightly run is a backstop for the AI model's behaviour drifting with no code change. Real calls catch that drift too, just after a customer sees it. |
| Voice prompt drift | Nightly 22:17 | Checks the live voice agents are running exactly the prompts and settings in our code, and that no unknown agent is live. | Reads the live voice agent settings. No writes, no calls. | Failed | No AI. Free. | only coverage Real calls can't tell you a live agent quietly differs from the code; they just behave differently. |
| Renewal harness | Nightly 03:00 | Runs the renewal workflow's states, signals and race conditions end to end. | Runs against production on the test property. Only cells that send no real texts, calls or emails. | Pass | No AI. Free. | likely only coverage Renewal auto-start is paused, so real traffic barely exercises renewals today. |
| Harnesses | Nightly 03:17 | About a dozen regression harnesses over the code: the draft lane, the writers, the AppFolio loop (offline), the leasing desk, turnover intake, the one-call-one-email count. | No. Offline. | Failed (4 lanes red) | No AI. CI machine time only. | Catches code breaks after merge. Real calls cover the end result but not the specific code path. |
| Weekly eval pack + audit | Sun 02:00 | Area-by-area Clara eval packs (renewal, leasing, prospect, maintenance, routing, resident services), the topic classifier eval, a read-only production data check, and mutation controls. The SMS and email stress tests in this run are already switched off (they now only run on demand). | The data check reads production. Evals are mocked. | Failed (leasing, prospect-comms and resident-services packs red; rest pass) | Uses AI. Not measured. | Likely overlaps the nightly Clara behavior evals heavily. |
| Opt-out (STOP) canary | Tue 03:30 | Proves that when someone opts out, every send to them is actually blocked, including voice. | Production, test property. Every send it triggers is a blocked send, so nothing goes out. | Pass (Oct 27) | No AI. Free. | only coverage Compliance: real opt-outs are rare and a broken block is invisible until we text someone who said stop. |
| Voice full sweep | Mon 03:30 | The layered voice regression suite against the live voice agents, as simulated text conversations. | Live agents, test property. No phone calls, no writes. | Pass (Oct 26) | AI tokens only. Not measured. | Real calls cover voice every day. |
| Intent evals | Mon 03:00 | Checks the small AI that reads what a message is asking for still gets it right. | No. Mocked. | Failed (Oct 26) | Uses AI. Not measured. | Real traffic covers this. Also runs on PRs that touch it. |
| Renewal evals | Mon 03:15 | The heavier multi-turn renewal conversation gauntlet. | No. Mocked. | Failed (Oct 26) | About $2 per run. | likely only coverage of renewal conversations, since real renewal volume is near zero right now. |
| Knowledge replay | Mon 06:30 | Replays past prospect questions from onboarded companies against what Clara knows, and scores how many she could answer. | Reads real customer guest-card questions. No writes, nothing sent. | Ran OK (Oct 26: 28% answerable from what Clara knows, 35% with the intake sheet) | Uses AI. Not measured. | A measurement for onboarding, not a pass/fail test. Real calls don't produce this score. |
2. Code-only checks (5): never touch the live system
Real calls can't cover any of these, because they test the code and the test suite, not the product. No AI; the only cost is CI machine time.
| Test | When | What it checks | Last result |
|---|---|---|---|
| Time travel | Nightly 03:17 | Runs the whole unit suite with the clock moved (month ends, leap days and so on) to find date bugs before they go off. | Pass |
| Mutation controls | Nightly 03:41 | Deliberately breaks things to prove each safety guard still catches it. | Failed |
| Database escape sweep | Nightly 23:23 | Checks no test accidentally talks to a real database. | Failed |
| Advisory tests | Nightly 01:30 | The flaky-prone tests that were moved out of the PR gate. | Pass |
| Test-read map | Nightly 02:23 | Records which files each test reads, so PRs only run the tests they affect. | Pass |
3. Not tests: listed so nothing looks hidden, not counted
- Rest of the 02:00 nightly queue: the overnight code builder (empty, does nothing), a refresh of the saved login cookie the opt-out canary needs, a sweep of Smith's send log, and Smith checking his own posts against the rules.
- 07:00 morning queue: Smith housekeeping (repos, sessions, disk), the marketing draft, the spend digest, and the Oracle reading yesterday's real conversations. The Oracle reads real calls; it doesn't test anything.
- Overnight watchers and drift checks: shared-secret match, phone-config drift, login-cookie preflight, phone-number sync, metric freshness, the production record check behind spine health, CI cache and branch cleanup. Read-only or housekeeping.
- Daytime, not overnight: the voice regression watch runs at 09:00 against the live voice agents and is red (0/6 could not run).