Nightly & Scheduled Suite Audit

• The biggest problem is alerting, not tests. Every suite that emails on failure got fixed within ~a day of a red (EL drift, suppression canary). The two suites with no failure notification — Renewal Evals and Nightly Stress Tests — have been red for 10 consecutive weeks and 7 consecutive nights respectively, with zero response. And even email isn't enough: the renewal harness sent 6 identical failure emails over 6 nights before anyone acted.
• Renewal Evals (weekly, Mon): red on all of its last 10 scheduled runs (~2 months). Deterministic — the same 5 live multi-turn tests fail every week ("renewal tool never fired"). Likely a broken fixture (tenant resolves with no propertyId → lease/balance lookups fail) — hypothesis, not confirmed. Either way the gate protects nothing and burns live Claude spend every Monday.
• Nightly Stress Tests: zero green nights this week, and the SMS lane has returned zero data for 5 straight nights — every scenario times out waiting for a Clara reply, the job hits its 120-min cap, and GitHub renders the run as "cancelled" (not "failed"), which is probably why nobody noticed. The one healthy run (Sun Jul 19) caught 7 real reschedule-tour regressions that remain unaddressed.
• Vacuous green is endemic. Four separate "passing" checks currently verify nothing: the renewal harness's prod-history replay job (exports 0 histories, then passes), WO Creation E2E and Pipeline Lab Evals (0 of last 30 runs each actually executed their harness — no Vercel preview → skip → green), and the stress-test timeout rendering as neutral "cancelled".
• The good news: the 6-night renewal-harness streak was pure harness rot (stale assertions after an intentional behavior change, fixed Jul 23), no product regression; the ops-guard crons (DLQ alarms, secret parity, EL config drift, voice-comprehensive, CI nightly) are firing on time and green; and EL config drift proved its worth by catching 15 real duplicate workspace tools on Jul 19.

1 · Renewal Evals: a weekly gate that has been red for 10 straight runs, silently

The live-multiturn job (npm run test:eval:renewal, live Claude) has failed on every scheduled run since at least May 25 — 10/10, deterministic, the identical 5 tests each week, all with the same shape: the renewal tool never fires (accept, decline-confirm, term-pick, escalate-handoff, counter-offer). The logs show the fixture tenant resolving with no propertyId, making get_lease_terms / get_tenant_balance fail — a credible reason Clara never reaches the tool. Fixture-vs-real-regression is unsettled — treat "broken fixture" as the leading hypothesis (same class as the #3011 identity-spine fix), not a finding.

Why nobody saw it: PRs run only the promptfoo job (green 12/12), the failing job is schedule-only, and the workflow has no failure-alert step at all — the only one of the failing suites with none. Meanwhile it spends live Claude API money every Monday to produce a red nobody reads.

Fix: reproduce locally → stamp the fixture or confirm the regression; add the standard failure alert; or deschedule deliberately. Runs: 29734634698 (Jul 20), 29242546911, 28786221891, … back to 26401123596 (May 25).

2 · Nightly Stress Tests: 5-night SMS blackout disguised as "cancelled"

Since Jul 20, every SMS-stress scenario polls for a Clara reply for 11 minutes and gets nothing (attempts=214 … lastMatchCount=0 — reply is ABSENT, not late), the job runs into its 120-minute cap, and GitHub reports the run as cancelled — which reads as neutral, not red. Result: zero SMS regression coverage for 5 nights, ~75% of the workflow's ~160 job-min/night spent producing nothing, and no notification (the workflow has none; a comment claims "Agent Smith surfaces failures" — Smith has been down).

Root cause of the blackout is undiagnosed. Two candidates: the harness polls /api/conversations?limit=100 and filters client-side while the test property carries 1,351+ conversations (saturated window → replies exist but fall outside it), or the inbound pipeline genuinely broke ~Jul 20 (~40 PRs merged Jul 21 alone). Cheapest discriminator: post one synthetic inbound, then query DynamoDB directly for the conversation.

The last healthy run (Sun Jul 19 full sweep) proved the suite still earns its keep: it flagged 7 real failures — 6 in the reschedule-tour cluster (reschedule_tour never called; Clara books instead) — with clean quarantine hygiene and owners on older items. Those 7 are still unaddressed.

Two weekday eval gates are also red for the wrong reason: triage-routing 89.19% vs 93% and prospect-comms 60% vs 79% both show 0 failed / N errors — broken fixtures (the yml's own comments admit sibling: "maintenance" is stale post-split), counted as quality failures. These are the same two "unowned eval regressions" tracked in the CI-cost thread — reclassified by this audit as harness rot, not model regressions (suspect commit 0897bcb48 likely off the hook, unverified).