Nightly test suites: why we run them, and what each one should become

"Readers" means people, measured as replies, reactions, issue comments or closes by a person. Cost is compute over the last 30 days (Oct 5 to Nov 4, scheduled runs, job start to finish times the runner's minimum vCPU, so a lower bound) unless the cell says otherwise. No dollar figure for CI: the repo has no measured price per vCPU-hour. Subscription tokens per run: not measured, except for the Clara nightly.

Seven smaller scheduled checks (config and doc invariants, not tests) are in their own table at the end. Fourteen scheduled ops jobs (deploys, cleanups, watchdogs) and the production monitors are out of scope.

Why it exists, in its own config: "so every day has one real eval reading even when PR sweeps were budget-skipped." That reason still holds. The PR-time copy of the same sweep runs on a daily budget that goes to whichever Clara branch pushes first after midnight UTC.

Window is 14 days because the sweep's ledger keeps 14 days; 30 days is not measurable. Control: a Nov 4 swept PR is classified as a Clara PR by the same rule.

Record: 27 nights, never green

• 27 scheduled nights (Oct 9 to Nov 4): 25 red, 2 cancelled, 0 green.
• Of 226 red domain-nights, 165 (73%) are zero-failure ("strict") domains tripped by 1 to 3 cases that pass on other nights.
• 12 domain-nights graded nothing (killed at the time limit while both subscription accounts were rate-capped, or a crash) and still printed FAILED.
• Last night (Nov 4) 13 domains were red. Six of them by exactly one case on a zero-failure bar. One crashed. One was red only because of the triage eval defect (85.5% against a floor of 92; 95.7% without the 12 broken rows). One graded models prospects do not get.

Every failing case, labelled

284 distinct failing cases across the 27 nights. 184 flip (fail, then pass), 59 stay red from their first failure, 41 failed once with nothing readable after. Per-case results exist only for the 12 nights from Oct 22 (older artifacts expire after 14 days), so "first red Oct 22" usually means "red on the first night we can read".

The domains that are red most often, and why

27 of the 35 domains that have ever been red have no owner in the eval catalog. 18 more domains were red on 1 or 2 nights each.

The two broken evals (our changes, not Clara)

1. Triage, broken by #24935 (merged Nov 3, 21:34Z; red at the next night's run, Nov 4 03:49Z). The eval works out a default transfer mode but never passes it on, so every leasing case is told "the leasing ring list has nobody behind it", and Clara correctly takes a message. 12 cases went red together. Production is not affected: the live call route sets the mode itself.
2. Tenant-services gauntlet, broken by #24241 (merged Nov 1, 06:47Z). A grader file path is written relative to the wrong folder, so the whole pack crashes every night since Nov 2 and has graded nothing for 3 nights. Its issue (#24409) has 0 comments.

The six real Clara gaps (red 12 of 12 readable nights, hidden because a floor absorbs them)

No green-to-red night is readable for these six, so no introducing PR can be named.

Regression candidate, not confirmed: LM-off-returning-prospect (triage). Passed the 5 readable nights Oct 22 to 27, failed all 7 since Oct 28 (two-sided Fisher exact p ≈ 0.0013), on the production voice model. At a building with leasing switched off, Clara hands a returning prospect to the leasing specialist. Of 213 commits between the two nights, two touched the files the eval renders: #22813 (Spanish triggers, unrelated) and #22906 (a comment). The cause is not known; a two-version A/B is the next step and is in the Clara lane.

Stale eval, candidate: maint-human-for-non-maintenance (cross-sibling-pivot). On 12 of 12 nights the maintenance specialist sends a billing question to the Lease & Billing specialist. The case expects a transfer, citing a prompt file that does not exist on main. The prompt owner needs to say which behaviour is wanted.

What it caught, and what it costs

32 merged PRs reference a nightly-eval issue: 14 change Clara (for example #22318 a declined tour time never carries an invented reason, #22397 a closed office answers "are you open?" correctly, #22428 a move-in total must equal its parts, #22999 a cancelled tour gets a goodbye), 13 repair graders or fixtures, 4 are CI. All 32 merged Oct 25 to 31 from one agent push; none before or since. The domains stayed red anyway, because each fix clears one case under a zero-failure bar while another case flips.

Where the reds go

• One GitHub issue per failing domain, never closed by design (one green night is not a fix), so the list is cumulative: 33 open, 0 closed. No Slack post at all; its config says no alerts and no channels. It is not one of the 14 "nightly tests" posts in #alerts in the retro.
• The issue filer says "the 8-hour CI report reads issues". That report's schedule is commented out and it has never run on its own.
• Readers: 257 comments in total. 107 are the bot. 150 are under a shared agent login and are all agent working notes, written only Oct 25 to 31. None can be attributed to a person. 0 reactions. The four issues opened since Nov 1 have 0 comments.
• Slack (indexed WhatsApp mirror, 9,672 messages): 0 mentions of the suite by name. Control: "nightly" returns 132 lines. The last human mentions of a nightly eval are from July and August, about an earlier version.

What it should become

1. Fix the two broken evals (in progress).
2. Grade the models production runs, or drop the arm. leasing-response should grade the prospect drafting model, keeping Sonnet 4.6 only for voice rows. Same for the other 149 cases whose failing arm is not production's.
3. Report new failures, not domain verdicts. The per-case results already exist every night. Report what newly failed, newly passed and is still failing versus the night before. Under a zero-failure bar with one try per case, red is the normal state and says nothing. This changes no pass mark.
4. Give the reds a reader (founder call): arm the 8-hour report, or send the new-failure list to whoever owns Clara quality.
5. Fix or delete each of the 82 flaky cases (Gera's rule: never quarantine, never retry to green). Pass-mark values stay a founder call (held PR #22343, open decision b93085657).
6. The PR-time sweep, as built, adds little. Once item 3 lands it should report the same diff, so a PR's receipt reads "this PR newly broke X".

Why not weekly or delete: weekly loses the one thing it does that nothing else does, dating a break to one night (triage Nov 4, gauntlet Nov 2, tour-home Oct 21). Deleting it leaves 1.4% of Clara PRs with any eval reading.

Voice regression watch: retire

• Protects: it was the only scheduled check of the live voice agents' behaviour (6 vendor and handyman transfer and message scenarios). Since Oct 31 ElevenLabs answers "endpoint sunset" (HTTP 410), so none of the 6 can run.
• Case history (105 local reports over 104 nights, Jul 19 to Nov 4): 525 case passes, 10 real assertion failures, 57 ElevenLabs 500s, 30 times the 410, 3 ElevenLabs 400s, 5 unclassified. Real regression: unknown-caller-vendor-eta-message (Sep 4, 12, 17), fixed by #7058. Stale eval: unknown-caller-vendor-onsite-transfer (Sep 29 to Oct 4), asserted a retired tool, re-pinned by #13741. The three maintenance cases never failed a real assertion.
• Caught / cost: 1 Clara fix against at least 7 PRs keeping the watch alive (#5270, #9115, #13153, #13741, #14844, #16042, #21899). The move to the new ElevenLabs API (#24871) was closed unmerged Nov 3. About 535 simulations, ≈0.80M credits, ≈$130 (estimate from the repo's own per-simulation constant and the Nov 3 price).
• Reds go to: #alerts, an email to Gera, and a failed-run record, every morning. 5 posts in 14 days, 2 human replies, both asking to remove it.
• Becomes: retired, waiting on decision b93719697. Note for that decision: the "keep and repair" recommendation on it assumes simulations are exempt from the Nov 3 no-tests rule (the rule says they are not) and that voice-comprehensive runs weekly (it was disabled Nov 3).

Nightly bench evals: keep retired

• Retired Nov 1. Verified: the nightly banners for Oct 25 to Nov 1 list 9 jobs including all three; Nov 2 to 4 list 6 and none of them.
• Maintenance history (122 of 125 nights Jul 1 to Nov 1 have a result): 21/21 on one night only. Three handyman cases red on 87 to 90 of 95 nights, with the record saying "synced" and "fail" at once; the reason lived in a spreadsheet that was not read. 14 cases flip 10 to 30 times each; the 4 voice cases never failed. Turnover passed 2/2 on 21 of 22 nights. Morpheus was never 3/3.
• Caught: 10 product fixes among 16 PRs named after these jobs (for example #6089, #8249, #8627, #19296). Upkeep: 117 PRs carry their names.
• Delete the three leftover schedule entries once their last run records age out (about Jan 30, 2027).

time-travel: weekly, with an owner

• History from its issue (#18610, 26 bot edits). 35 files on Oct 10, 2 by Oct 12. Since Oct 23, four conversation-thread and email-card tests break at +1 year on all 13 runs, so main turns red on Jan 1, 2027. Likely trigger #21979 (Oct 23), matched by date, not bisected. One ledger test fails at +30 days on all 26 runs (another check watches the same thing). These are test bugs (fixtures dated 2026), not product bugs.
• Weekly is enough because its cells look 30 days to a year ahead.

ddb-escape-sweep: owner or stop

• Test files that reach the real database rose 3 → 28 → 49 (Oct 9) → 64 (Oct 12) → 72 (Oct 24) → 98 (Oct 31) → 112 (Nov 4). Since Oct 26 the 120-minute cap often stops it early (Nov 4: 9,309 of 10,752 files checked), so 112 is a floor.
• A test that can reach a real table is a real risk on a machine that holds production credentials. Either someone burns down the 112 (the issue names each file) and it goes weekly, or it stops.

harnesses: keep nightly, route each red

• 20 runs, 17 red, 15 in a row; last green Oct 20. By harness: portfolio red since Oct 21 (a stale saved grid); one call, one email red since Oct 28; 11 writer scenarios red since Oct 29; draft lane red since Oct 31.
• 15 of 38 job failures are a lost runner at the type-check step (cancelled 0.3 to 0.7 minutes in, on small spot machines), not a test result.
• The possible bug, ruled out: the one-call-one-email harness went red on Oct 28 because #22894 (Oct 27) moved every company onto the team-notification engine, and the test bench was never set up for it (its database store and building-mode reads have no local stand-in). Production was checked read-only: of 5 open-hours calls since Oct 28 at real customers where the transfer reached nobody, the team got a delivered email for 4; the 5th was skipped by the Oct 4 no-message rule. So the harness has not really tested the engine path for any call shape since Oct 28; repairing it is its own lane.

nightly.yml: shrink to the 10 packs only it grades

• 42 eval configs; 32 also run in the nightly Clara sweep. Ten are deliberately left out of the sweep because they have no sweep pass mark (clara-application-link, clara-escalation-honesty, end-conversation-sms, identity-router, outreach-copy, outreach-price-drop, over-escalation, pm-english, renewal-scoping, renewals-sms). This weekly run is their only scheduled grading.
• Readable runs: Oct 25 (65 failing rows of 608) and Nov 1 (129 of 593); 38 cases failed both nights.
• Possible real gap: the same 5 of 28 over-escalation cases failed on both nights, with Clara answering off-menu lease terms with "I'll check with the leasing team and get back to you", which Gera's Oct 28 rule forbids. Whether Sonnet 4.6, the model it was graded on, is the model that live reply path uses is not settled: the two research passes read different model tables. Also one Spanish case answered in English (real miss); outreach-copy and price-drop expect the name "Clara" that production strips (stale).
• Moving the 10 into the nightly sweep would need new pass marks, a founder decision (#22343). So: one weekly job for those 10, delete the 32 duplicates and the duplicated mutation and ledger steps.

The rest

• advisory-tests: green 16 nights since Oct 20. It holds tests that were moved off the PR lane for being timing-flaky; the rule is fix each so it is deterministic and return it to the unit lane, or delete it.
• renewal-evals: the red job is the live multi-turn renewal eval, on 3 of the last 5 Mondays (case detail not measured). Its header says nightly.yml covers renewals daily; false since Oct 4.
• intent-evals, voice-prompt-drift, mutation-control-sweep: cheap and mostly green; keep, with an owner for the red.
• goldmine-live-scorecard, ingest-replay-weekly: product metrics posted where nobody answers; move them to a dashboard. ingest-replay is not on the schedule list at all.
• semgrep: the only security scan, off since Sep 30 while the schedule list still shows it. A founder or security call.

The seven scheduled checks

• Retire the voice regression watch: decision b93719697, open since Nov 3.
• Give the reds a reader: arm the 8-hour CI report, or name an owner per suite. Arming it is marked founder-gated.
• Pass-mark values for the eval domains: held PR #22343 and decision b93085657.
• semgrep: back on, or deleted.

Everything else on this page is agent work. A board was set up on Nov 1 to retire overnight tests; it still holds only its template rows and nobody is driving it.

• Dollars for CI compute (no measured price per vCPU-hour), and AWS cost of the Clara nightly's runners.
• Subscription tokens per run for every suite except the Clara nightly.
• Case state on Oct 9 to 21 for Clara domains that passed on a floor (results expired).
• Whether leasing-response would pass on the models prospects actually get.
• The cause of LM-off-returning-prospect (no A/B run yet).
• nightly.yml case history before Oct 25 (expired); the renewal live job's failing cases; the maintenance handyman failure reasons.
• Email opens for the voice watch; any reading outside GitHub and the indexed Slack.
• PR-time execution of renewal and intent evals comes from a 60-run sample on Nov 4 afternoon, not 30 days.