Clara on Claude 5.5: upgrade, release, and proof

Clara today (measured)

From the assembled SMS leasing prompt, built with a synthetic identity: 33a316db-d58f-4e01-a7e6-64aebd1b7e79/scratchpad/modernization/assembled-sms-leasing.txt (tools in assembled-sms-leasing.tools.json). Approximate.

Q1. How instruction following changed (4.x to 5.5)

• Since Opus 4.5/4.6, models "may now overtrigger. The fix is to dial back any aggressive language": "CRITICAL: You MUST use this tool when..." becomes "Use this tool when..." [5].
• Sonnet 5 "interprets prompts literally... does not silently generalize an instruction from one item to another, and it does not infer requests you didn't make" [6]. The Sonnet 5.5 guide keeps those patterns as "a reasonable starting point" [1].
• Give the reason with the rule: "Claude is smart enough to generalize from the explanation" [5].
• Say what to do, not what to avoid [5]. A needless prohibition "can *anchor it toward* that failure" [8]. [S] A 2026 paper finds naming a forbidden word primes it (models unnamed) [36].
• Use "diverse, canonical examples", not "a laundry list of edge cases" [4]; 3-5 varied examples [5].
• Many CRITICAL markers stop carrying information, and "an anxious prompt produces a cautious, hedging model" [8]. Incident patches become "a maze of special cases" that "fails unpredictably between them" [8].
• Rule count, 4.x data only: Claude Sonnet 4 followed 94.4% of instructions at 100 and 42.9% at 500, mostly by omitting some and favoring earlier ones [26, S]. Clara's 175 all-caps markers plus hundreds of softer rules fall in that range; 5.5 is unmeasured.
• Implication: rewrite the tone and fold patches into principles; keep product context and the reasons.

Q2. Autonomy versus hard-coded flows and forced tools

• Anthropic's default is the simplest design that works: workflows for predictable paths, agents for open-ended ones [9].
• Sonnet 5.5 rejects tool_choice any and tool with a 400, token counting included [3]. Haiku 5.5 accepts them, but a forced call skips thinking [10].
• Replacement: auto plus strict: true, with the prompt saying when the tool applies [3]. Because auto "does not guarantee a call, check that one was made and retry if it wasn't" [7, P-skill]. On Bedrock (no strict tool use for Sonnet 5.5), validate in code [11].
• Better: "count the model-call sites and ask of each whether its inputs fully determine its output" [8]. Clara's prompt says "CRITICAL: Call save_prospect with their name, phone ... and property_id ... as your VERY FIRST tool call", with every value already filled in by the code. That step belongs in code.
• Tool descriptions are "by far the most important factor in tool performance"; aim for "at least 3-4 sentences" [10]. Contract in, behavior steering and examples out, and the system prompt "shouldn't name tools" [8]. When-to-use lives in the tool's description; a turn that needs a specific tool gets a mid-conversation system message after the user turn [12].
• Sonnet 5.5 batches tool calls more than Sonnet 5 [13]. When it miscases a tool name, accept unambiguous matches or return is_error with the exact name [1].
• "Behavioral rules that could be hooks, allowlists, or schema validators are less reliable as prose" [8].

Q3. Thinking and effort

• Thinking is on by default on both models [14]. Sonnet 5.5 rejects thinking: disabled; its lowest setting, between_tools, works only at high or below and blocks per-turn effort changes [3].
• Defaults: Sonnet 5.5 high, with levels "recalibrated", so re-sweep [3]; Haiku 5.5 medium [2].
• Sonnet 5.5 chat: "start at medium or low". "From medium up, the model thinks briefly before almost every reply, even a greeting... Asking it in the system prompt to think less doesn't reliably reduce its thinking. At low, it skips thinking on most simple requests" [1].
• Haiku 5.5: low for chat and volume, high for "strict instruction following". Telling it to answer directly "didn't stop it from thinking" [2].
• Hidden thinking speeds the first word, not the bill [14].
• Pitfalls: leave room in max_tokens for thinking; structured output at low/medium can think until max_tokens (count it as failed); asking for reasoning in the reply invites reasoning_extraction refusals [1]. On Haiku 5.5, thinking off plus JSON output can skip a needed tool call [2].
• Changing top-level effort breaks the cache; the per-message effort beta keeps it [1].
• If the voice platform calls Claude for us, check that it exposes these settings (not verified).
• Implication: voice and SMS start at low, email at medium. Measure p50/p95 time to first word; raise effort only where evals show a gain.

Q4. Caching, knowledge, long context

• The cache matches the request prefix byte for byte: tools, then system, then messages [12]. Minimum cacheable prompt is 512 tokens on both 5.5 models (4,096 on Haiku 4.5) [3][16]. Reads cost 0.1x input, 0.05x on Sonnet 5.5 ($0.10 per million tokens) [17].
• Editing tool definitions invalidates everything; changing effort or thinking settings invalidates the messages [15].
• Haiku 5.5 guide: "Render the date once, when the conversation starts"; on a later day, "give the new date in the newest turn" [2].
• Clara's date sits 4% in and per-person details 11% in. So at most the first ~11% of the system prompt can be shared across conversations; the other ~89% is processed fresh for each new conversation, and the date resets even the shared part daily (my inference from [12][15]; whether Clara sets cache_control today is not checked).
• Haiku 5.5 price step: $0.10/$0.50 per million tokens up to 100K prompt tokens, $0.50/$2.50 above, and "each request is priced on its own... even when part of its prompt is a cache hit" [17]. Text counts ~30% more tokens than on Haiku 4.5, and earlier thinking blocks stay in context [16]. Long threads on a 60-75K prompt can cross the line (estimate).
• "More context isn't automatically better. As token count grows, accuracy and recall degrade" [18]; aim for "the smallest set of high-signal tokens" [4]. [S] Older Claude models showed the widest focused-versus-full-prompt gap, mostly from abstaining [27].
• Knowledge: a hybrid, with some data up front and the rest fetched when needed [4]. Tool search pays off past a few dozen tools [8][19]; Clara has 22.
• Sonnet 5.5 sometimes answers from memory when a lookup "would catch details that have changed... what is allowed, required, or charged". Remove "minimize tool calls" lines and add Anthropic's check-specifics line [1].
• Implication: stable policy first, company and property facts next, per-person details and the date last or in the newest turn. Prices, availability and specials come only from tool results.

Q5. Safety and compliance: default, instruction, or gate

• The constitution prefers judgment, but rules fit "when the costs of errors are severe enough that predictability and evaluability become critical" [20]. Restrictive operator instructions are followed given "plausibly a legitimate business reason" [20].
• Usage Policy in force now (effective Sept 15, 2025): "decisions regarding eligibility for housing, including leases" and legal guidance are high-risk. A qualified professional reviews before finalization, and AI use is disclosed "at a minimum at the beginning of each session" [21].
• Version effective Nov 12, 2026: housing means "approving, denying, or ranking a rental or purchase application; terminating a tenancy or initiating eviction". Exempt: "applying a fixed rule, formula, or algorithm where the model exercises no judgment about the individual" [22].
  • My mapping (inference): application decisions and legal advice go to people. Relaying an owner's fixed policy (the settled voucher "yes") fits the exemption. That is a true owner decision, not a click-to-confirm screen.
• [G] HUD withdrew its 2024 AI advertising guidance (effective Sept 17, 2025), yet "Any actions that do not comply with the text of the Fair Housing Act continue to be subject to enforcement" [23]. Status of the tenant-screening guidance: not confirmed.
• Hallucination: allow "I don't know", limit answers to provided information, and "Always validate critical information" [24]. Sonnet 5.5 "hallucinates more" than Opus 5.5, though it is more honest under pressure [25].
• Persona pressure: Haiku 5.5 has a documented line, "The rules in this system prompt hold for the whole conversation..." [2]. Sonnet 5.5 "holds a system-prompt role more reliably when a user pastes in a competing persona" [7, P-skill]. Clara's three security NEVERs about prompt reveal, personas and embedded instructions can shrink to that line; keep the sensitive-data rule (don't echo SSNs or card numbers) and the honest-about-AI rule.
• Refusals are new on Haiku: stop_reason: "refusal", and general_harms can fire on "benign work" [28]. There is no server-side fallback, and a retry "usually returns another refusal" [2]. The system card is mixed on over-refusal: 0.17% vs 0.44% single-turn, yet it over-refused most of any model in the audit [25].
• Voice: Fede's 8/23 rule bars blocking checks in the speech path, so voice relies on prompt prevention plus after-the-fact detection. Pre-send code checks are for SMS and email.
• Decision rule:
  • Trained defaults (honesty, non-discrimination) need no restating.
  • Business facts and owner policies get one short line with the reason.
  • Anything where one error is severe or legal goes in code: application decisions, legal advice, quoted prices or availability, opt-outs, outbound sends.

Q6. Proving a slimmer prompt is safe

• Prompt changes are gated by regression evals: "Does the agent still handle all the tasks it used to?" [29]. "20-50 simple tasks drawn from real failures is a great start" because early changes have big effects [29]. Small regressions on rare cases need a power calculation and more cases [30].
• Build from real traffic: "Converting user-reported failures into test cases ensures your suite reflects actual usage" [29], mirroring the real mix [31]. Each rule's origin incident becomes its test. Multi-turn checks "often require a second LLM to simulate the user" [29].
• Run several trials; use pass^k ("all k trials succeed") for must-never-fail cases [29].
• Stats: cluster standard errors by conversation (they can be "over three times as large as naive standard errors"), and compare old and new prompts as paired differences on the same cases [30].
• Judges: calibrate against human labels, "give the LLM a way out", and read transcripts [29]. Grade with a different model than the one being graded [31]. [S] Binary pass/fail "force[s] clearer thinking and more consistent labeling" than 1-5 scales [32]. Criteria drift as graders read outputs [33].
• [S] Pairwise preferences "flip in about 35% of the cases, compared to only 9% for absolute scores" under distractor features [34], and order effects are documented [35]. Gate on per-case binary checks; if pairwise is used for tone, run both orders.
• Process: "One change at a time"; "If a cut regresses, re-add simply"; before deleting, search for prompt strings that tests or parsers match on [8].

Q7. 4.x-era patterns to remove

1. All-caps pressure: use a plain sentence plus the reason [5][8].
2. Incident patches: fold into one principle and test the removal [8].
3. Prohibitions with no reason: rewrite as positive statements; keep business and compliance ones [5][8].
4. Forced tool_choice and missing-call retry loops: use auto + strict, or code [3][8].
5. Prose that steers thinking, and reasoning fields in output: use the effort setting [1][8].
6. Tool-discouraging lines ("only use tools when strictly necessary"): remove [1].
7. "Don't narrate" lines and scheduled reminders: remove, or use turn-scoped messages [8][12].
8. Numeric word caps: describe the audience instead; real SMS format needs stay as format rules [8].
9. temperature, prefill, budget_tokens: remove (400 errors on both 5.5 models) [11][16].
10. Mid-conversation edits to system, tools or history, and user text in tool_result: keep conversations append-only [3][1].
11. Behavior rules in tool descriptions, and tool names in the system prompt: rewrite descriptions as contracts [8][10].
12. One gold example or stale examples: use 3-5 varied ones [5][8].

Keep: product and audience context, reasons, compliance wording, exact scripts for fragile steps, tool contract detail, prohibitions whose failure still reproduces, and a one-line role [8].

Verification notes

• Read word for word against downloaded page text: [1]-[8], [10]-[18], [21], [22], [25] (the quoted sentences), [29], [30], [32], [34], [36] (abstract only).
• From research subagents' fetches (spot-check before publishing): [9], [19], [20], [23], [24], [26]-[28], [31], [33], [35].
• [7] and [8] are Anthropic-authored but live in the skills repo, not the platform docs.

Citations

1. Prompting Claude Sonnet 5.5. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5-5 [P]
2. Prompting Claude Haiku 5.5. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-haiku-5-5 [P]
3. What's new in Claude Sonnet 5.5. https://platform.claude.com/docs/en/models/sonnet-5-5/whats-new-sonnet-5-5 [P]
4. Effective context engineering for AI agents (Sep 29, 2025). https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents [P]
5. Prompting best practices. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices [P]
6. Prompting Claude Sonnet 5 ("More literal instruction following"). https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5 [P]
7. claude-api skill, model-migration.md (Sonnet 5.5 and Opus 5.5 sections). https://github.com/anthropics/skills/blob/main/skills/claude-api/shared/model-migration.md [P-skill]
8. claude-api skill, prompt-audit.md. https://github.com/anthropics/skills/blob/main/skills/claude-api/shared/prompt-audit.md [P-skill]
9. Building effective agents (Dec 19, 2024). https://www.anthropic.com/engineering/building-effective-agents [P]
10. Define tools. https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools [P]
11. Claude Sonnet 5.5 migration guide. https://platform.claude.com/docs/en/models/sonnet-5-5/migration-guide [P]
12. Mid-conversation system messages and tool changes. https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages [P]
13. Introducing Claude Sonnet 5.5 (Sep 28, 2026). https://www.anthropic.com/news/claude-sonnet-5-5 [P]
14. Thinking. https://platform.claude.com/docs/en/build-with-claude/thinking [P]
15. Prompt caching ("What invalidates the cache"). https://platform.claude.com/docs/en/build-with-claude/prompt-caching [P]
16. What's new in Claude Haiku 5.5. https://platform.claude.com/docs/en/models/haiku-5-5/whats-new-haiku-5-5 [P]
17. Pricing. https://platform.claude.com/docs/en/about-claude/pricing [P]
18. Context windows. https://platform.claude.com/docs/en/build-with-claude/context-windows [P]
19. Introducing advanced tool use (Nov 24, 2025). https://www.anthropic.com/engineering/advanced-tool-use [P]
20. Claude's constitution. https://www.anthropic.com/constitution [P]
21. Usage Policy, effective Sept 15, 2025 (archived copy; in force until Nov 12, 2026). https://www.anthropic.com/legal/archive/22742366-2ef0-4c7a-a833-6523f10d3944 [P]
22. Usage Policy, effective Nov 12, 2026. https://www.anthropic.com/legal/aup [P]
23. HUD FHEO guidance withdrawal notice, FR-6571-N-01, https://public-inspection.federalregister.gov/2026-06624.pdf; HUD's 2024 AI guidance release, https://archives.hud.gov/news/2024/pr24-098.cfm [G]
24. Reduce hallucinations. https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations [P]
25. Claude Sonnet 5.5 and Claude Haiku 5.5 system cards (Sept and Oct 2026), indexed at https://www.anthropic.com/model-card [P]
26. Jaroslawicz et al., "How Many Instructions Can LLMs Follow at Once?" (IFScale, Jul 2025). https://arxiv.org/abs/2507.11538 [S]
27. Chroma Research, "Context Rot" (Jul 14, 2025). https://www.trychroma.com/research/context-rot [S]
28. Refusals and fallback. https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback [P]
29. Demystifying evals for AI agents (Jan 9, 2026). https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents [P]
30. A statistical approach to model evals (Nov 19, 2024). https://www.anthropic.com/research/statistical-approach-to-model-evals [P]
31. Define success criteria and build evaluations. https://platform.claude.com/docs/en/test-and-evaluate/develop-tests [P]
32. Husain and Shankar, "Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings?" (May 29, 2025). https://hamel.dev/blog/posts/evals-faq/why-do-you-recommend-binary-passfail-evaluations-instead-of-1-5-ratings-likert-scales.html [S]
33. Shankar et al., "Who Validates the Validators?" (Apr 2024). https://arxiv.org/abs/2404.12272 [S]
34. Tripathi et al., "Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation" (COLM 2025). https://arxiv.org/abs/2504.14716 [S]
35. Wang et al., "Large Language Models are not Fair Evaluators" (May 2023), https://arxiv.org/abs/2305.17926; Zheng et al., MT-Bench (Jun 2023), https://arxiv.org/abs/2306.05685 [S]
36. "Semantic Gravity Wells: Why Negative Constraints Backfire" (Jan 2026; abstract only, models not named). https://arxiv.org/abs/2601.08070 [S]

Status: finished for Fede, 2026-10-16 (data up to about 03:40 UTC Oct 17). Read-only: repo main as of 4bc9c975 (Oct 17, 03:08 UTC), GitHub Actions history, the docs site. Nothing was run, changed or posted.

How to read the numbers: each number names its source in brackets. ESTIMATE = extrapolated from a sample. INFERRED = a judgment from the evidence, not a measurement. Bracketed file names are the session's working files; the findings they hold are reproduced in this report.

Answer first

1. They run a lot. Nothing listens to them. The main Clara suite was started by hand 586 times between Oct 6 and Oct 14 (412 of those on Oct 10-11), on top of a daily sweep on pull requests and a nightly sweep [workflow 310386787 run list]. No Clara behavior eval can stop a merge. The required checks are Build, Type Check, Unit Tests, Lint, review, Nested Gate and "Regression Gate — Leasing (required)" [branch protection and ruleset API]. That last one scores an 18-case side package and skips itself on every Clara change: "0 fixtures run, 0 domains scored" [ci.yml line 6210].
2. The main suite has been red on every full run since Oct 7. 16 of 16 full sweeps were red, Oct 7 to Oct 17 UTC (15 on pull requests, 1 on main), and all 15 pull requests merged anyway. 8 of 8 nightly sweeps were red (Oct 9 to Oct 16). The 14 issues the nightly opened are all open; the only comments on them are the bot's own "Still red" [sweep ledger; gh pr list; issues #18909 to #19479].
3. Mostly the tests are broken, not Clara. Five topic suites ("domains") fail almost every night, three of them on all 8 nights. All five turn red on a single bad row because they have no measured pass bar. One grades a model production does not use for that job, one grades as text what production does by voice, and one had its harness edited twice that week. One small real gap: staff chat says "I can't send texts directly". One unexplained change worth a look: since Oct 11, in the plain after-hours case, the voice prompt answers "yes, we're open right now" (section 3).
4. "Multi-turn" mostly means a fixed script with one graded reply. Nothing puts an AI-played renter against Clara's real tool-running loop. The real catches of the last 60 days came from replays of real records and live runs on Willows, not from the scripted suites (section 5).
5. Money is small; attention is the cost. Eval jobs are about 6% of pull-request CI job time, roughly $60 a month of machines (ESTIMATE). One suite still bills a paid API key on every change to the email extractors, against the subscription-only rule (section 7). The bigger cost is a signal that is always red, which teaches everyone to ignore red.
6. Recommendation: keep a small core and hang it on the testing ladder. Delete 8, merge 8 into others, rebuild 2 on the sealed replay and simulation sandbox being built tonight (evals/model-55-replay, not on main yet), keep 16 (section 8).

1. Inventory

1a. What each suite is

"Real model" = a live Claude or ElevenLabs call happens. "Turns" = how many turns the test drives by itself.

The index of all this, the eval catalog (src/lib/evals/catalog.ts), lists 130 entries: 103 promptfoo, 16 vitest, 9 harness, 2 Temporal bench. 70 are graded by an AI judge, 60 by fixed checks. 51 are manual only. 98 have no owner. 43 pin a fixed model instead of the production tier. 27 of those are flagged because the pinned name cannot follow the production tier: a Sonnet 4.6 pin matches production today only by coincidence, and a Haiku pin grades the wrong model now. Its header says "NOTHING READS gate YET", so the 130 proposed pass bars are read by nothing; 73 entries record a bar that some runner actually enforces [catalog.json]. An admin page (admin/dev/evals) shows each entry and its last stored CI result; whether anyone opens it is not measured. 16 of the 103 promptfoo configs are in no sweep and no workflow: 13 can be started by hand through a subscription runner, and 3 (first-email pet labelling, named-home filtered, property aliases) have no runner at all [static scan, verify/cfgrefs.py].

1b. How each suite runs

PR-lane numbers for rows 1-13 come from 192 sampled CI runs (12 a day, Oct 1-16) out of 11,577 [ci/jobs/*.json, ci/lane-sample.txt]. Until Sep 30 each suite was its own workflow; since Oct 1 they are planned lanes inside one CI workflow, so the standalone history stops at Sep 30. "Red, recent" uses that standalone history unless it says otherwise.

Typical run = median wall clock from run-summary.txt (30 days) or job-minutes per run in the CI sample; bill = which account pays for model calls. Sources: [runs/<workflow id>.json, run-summary.txt], [ci/sweep-results-artifacts.json, ci/sweepjobs/*.json], [ci/red-branches.json, ci/red-branch-prs.tsv], [verify/stress-p*.json, verify/harn-sched.json], [branch protection and ruleset API], [ci.yml lines 1655-1667].

2. Never run, or not run in 30+ days

Also dead weight: 53 committed nightly logs from Mar 25 to Jul 6 in evals/gauntlet/; the pm-reminders harness was never wired to anything (its own marker says so); 8 of the 103 subscription runners are referenced by nothing [inventory notes].

3. Red right now

Also worth knowing: Intent Evals' last weekly run was green (Oct 12), but all 5 manual runs on main in 30 days were red. The latest (Oct 15) failed 1 of 54 cases: a proposed tour did not confirm on a genuine "yes" [Oct 15 log].

Why the Clara nightly is red [8 nightly runs, Oct 9-16: artifacts and logs in nightly-rca/; matrix from nightly-rca/tools/matrix.py]:

• It is not infrastructure. Every night all 4 jobs finish, the longest takes 14.6 of its 30 minutes, and there are zero timeouts. One night (Oct 13) two domains failed only on "Connection error."
• 6 to 9 domains fail per night; 16 different domains failed at least once. Five fail almost every night. All five are "strict, no measured baseline": one failing row turns the whole domain red.

Plus leasing-on-behalf is a coin flip: the same commit, dispatched three times, split 1 pass and 2 fails, twice.

Verdict on "model drift, stale fixtures, or real regressions?": mostly stale tests and wrong pass bars, one wrong model, one minor real gap, one unexplained flip that deserves a look. No infrastructure failures.

4. Overlap

5. What they caught, and what nothing caught

Caught in the last 60 days (keyword search of 11,264 PR titles and bodies; not exhaustive) [ev/cand_partial.json, ev/prs.jsonl]:

From the Clara suites themselves: one small prompt fix. #18853 (Oct 12) started from the on-behalf domain being red on main: it made that grader stricter and added worked examples so Clara's refusals to a visitor stop listing another person's application, unit or status. Two other eval PRs fixed the tests, not Clara: #17452 (sweep split into 4 jobs so it could finish) and #18232 (a scripted tool call lost its result). Otherwise, 15 of 15 red-sweep PRs merged and 14 of 14 nightly issues sit untouched.

Not caught by anything before a customer saw it. Source: 15 incident write-ups on the docs site, Aug 18 to Oct 16 [inc/*.txt], plus two incidents recorded in Fede's rules file. 13 of the 15 pages never mention an eval. Whether a suite could have seen each one is INFERRED:

Not in the table: two pages were merged into the release hub and not read (the Aug 29 tour-day dead end, the Aug 30 escalation false trigger), and one page is a code audit, not an incident (Oct 16 supervised mode).

INFERRED takeaway: the scripted suites grade one model reply in isolation. The faults that reached customers lived in gates, routing, telephony, data and multi-day flows. Those were caught, when they were caught, by replays of real records and live runs on Willows.

6. Multi-turn: real simulation or fixed replays?

Answer: nothing puts an AI-played renter against Clara's real tool-running loop. The full-loop harnesses use fixed or recorded renter turns and run by hand. The true simulations are either voice with mocked tools and report-only, or ran once. That is the gap the sealed replay and simulation sandbox is meant to fill [inventory notes; scripted-tool-results.ts lines 42-44; run-renewals.ts header; voice-comprehensive header; replay-harness README].

7. Cost

History behind today's guards: on Jul 5, agents running plain unit tests "burned ~$600 of production-key credits in a day", which is why live-model suites are opt-in [vitest.config.ts lines 105-108].

Side effect worth knowing: per the testing playbook, the CI Clara eval run wipes every open work order on Willows around 03:40-04:30 UTC, so live runs there must avoid that window.

The expensive part is not money. It is a suite that is always red, so nobody reads it, plus 586 hand-started runs in 9 days that each served as someone's merge proof, half of them red.

8. Recommendation per suite

Totals: delete 8, merge 8, rewrite on the sealed sandbox 2, keep as gate 3, keep as nightly or weekly signal 5, keep as on-demand or rung-4 tools 8.

9. Place in the stack

Today the testing playbook's ladder (rungs 1 to 6) never mentions the promptfoo suites; they sit beside it. Proposal: every Clara behavior change (prompt, tool, gate, loop, model) runs these, in order:

 rung 1   unit test + planted defect           (required today)
   +      the touched domain's scripted cases  (green, measured floor, production tier)
 rung 2   sealed replay of real conversations  (evals/model-55-replay; renewal replay for renewal paths)
 rung 3   simulated renters, real loop, sealed (simulation rung of the same sandbox)
 rung 4   Willows live run on the real path    (coworker, Willows harnesses, maintenance grid, robot call)
 ------
 nightly  full domain sweep as a trend line, owned issues, never a merge gate

Three rules that would make this stick: a suite that has been red for 7 days gets fixed or quarantined, never left red; every suite has an owner or is deleted; a suite grades the production tier, never a pinned model.

Sources

• Actions run lists per workflow, pulled Oct 16: runs/<workflow id>.json, run-summary.txt; full Nightly Stress Tests history and Harnesses Sunday runs: verify/stress-p1.json, verify/stress-p2.json, verify/harn-sched.json.
• CI pull-request runs Oct 1-16 (11,577) and the 192 sampled job lists: ci/day-*.json, ci/jobs/*.json, ci/lane-sample.txt, ci/analyze.py.
• Completed sweep ledger (19 artifacts, 16 runs) and per-job outcomes: ci/sweep-results-artifacts.json, ci/sweepjobs/*.json; PR states checked with gh pr list --head.
• Red-run vs merge cross-check (113 branches): ci/red-branches.json, ci/red-branch-prs.tsv.
• Branch protection and rulesets: gh api .../branches/main/protection and .../rules/branches/main.
• Eval catalog dump: catalog.json from src/lib/evals/catalog.ts; config wiring scan: verify/cfgrefs.py.
• Nightly root cause (8 nightly runs, artifacts, logs, issues): nightly-rca/interim-cbe.txt, nightly-rca/issues.json, nightly-rca/tools/matrix.py, nightly-rca/rh-logs/, nightly-rca/stress-jobs/, nightly-rca/quick/.
• Harness inventory notes: notes.md, eval-inventory-notes.txt.
• Catch candidates (keyword search of 11,264 PRs): ev/cand_partial.json, ev/prs.jsonl.
• Incident write-ups (15 docs pages, Aug 18 to Oct 16): inc/*.txt.
• Last commits per asset: last-commits.tsv.
• Docs: testing-playbook, eval-inventory-2026-08-20, clara-eval-coverage-audit, agent-testing-cadence-2026-09, ci-cost-study-2026-10, ci-runner-cost-2026-10.
• Repo files read: the Clara, intent, renewal, voice, Pipeline Lab, nightly, renewal harness, Harnesses, eval-diff and closer workflows; ci.yml; evals/run-all.sh; evals/README.md; docs/evals/README.md; quality-gate-poc/README.md; evals/gauntlet/run-renewals.ts; evals/fast-tier/README.md; scripts/trace-replay/replay-intent.ts; evals/replay-harness/README.md; src/lib/evals/scenario-detail.ts; vitest.config.ts.
• Not finished (the helper agents hit their rate limit at about 03:50 UTC Oct 17): an exhaustive 60-day catch list, and the failing jobs in the Harnesses Sunday runs.