Agent Testing Cadence

1. The question

Adding touring and enrichment logic is like adding new rooms to a house while people still live in it — each room is fine alone, but the hallways between them are where things get missed.

• New touring and enrichment logic means more moving parts talking to each other.
• Each part can pass its own test and still disagree with its neighbor in production.
• The fear: nobody would notice until a real caller hit the seam.

Proves: the shape of the question Fede asked, 2026-09-13.

2. What runs today, by cadence

It's not an empty gym — it's a gym where the machines that work on individual muscles are booked solid, and the ones that test whether your whole body moves together are unused.

• 27 GitHub jobs already run nightly or weekly — voice, SMS/email, renewals, config drift.
• The full CI suite also reruns every night, not just on each pull request.
• Correcting the premise: CI is not silent — the gap is narrower and specific.

Proves: propflow-testing-inventory-2026-09-13.md §1–2, full workflow list in the appendix.

Deeper → Appendix: every scheduled workflow

3. The three real gaps

Like a smoke detector with a dead battery — it looks installed, the light even used to blink, but nobody would know it stopped working until there's smoke.

• Coworker-facing scenario harness: fires only on a diff to one file, no timer.
• Mac nightly eval script: header claims 4am, no matching schedule exists anywhere.
• PR gate: missing one key means live config-drift checks quietly skip, not fail.

Proves: inventory §2 (coworker), §2 (Mac script, no plist in ~/Library/LaunchAgents), §4/drift audit (PR-gate key gap).

4. How the field does it

Same four checkpoints an airline runs before a flight: pre-flight checklist, night-before maintenance, a black-box recorder while flying, and a test flight before the plane ever carries passengers.

• Sierra's tau-bench checks agents succeed repeatedly, not just once.
• Decagon runs the new agent silently alongside the live one before switching over.
• ElevenLabs and Cresta simulate full multi-turn conversations, not single replies.
• SRE teams run one small fake transaction through production every few minutes.

Proves: agentic-testing-industry-research-2026-09-13.md §1–5, with source links in the appendix.

Deeper → Appendix: industry sources table

5. The pairs that actually break

Two coworkers can each be great at their job and still hand off work wrong — the failure isn't either person, it's the handoff.

• Each pair needs a check on the handoff, not just on each side alone.
• Voice-prompt-to-live-config already has one (nightly); it's the model to copy.
• The other four pairs don't have a dedicated check yet — see the appendix table.

Proves: inventory §5 gaps table, reproduced in full in the appendix.

Deeper → Appendix: component-pair gaps table

6. Proposal: a tiered cadence

Like a restaurant kitchen: taste every dish before it goes out (per PR), do a full menu run-through before the doors open (nightly), and have one dish sent to a real table every day just to make sure the kitchen still works (canary).

• Per PR: deterministic checks plus a prompt/tool config snapshot diff.
• Nightly, on the subscription runner: 25–50 golden journeys, one full robot call/text/email each.
• Continuous: one synthetic call and text per live line daily, alerted through Sentry.
• Weekly: the adversarial red-case set.

Proves: industry research §6–7 (tiered pattern, 2-property sizing), inventory's identified gaps this closes.

7. Cost and token discipline

Same rule as any recurring bill — automate it on the plan you already pay for, don't open a new metered tab every night.

• All new nightly/continuous jobs run on the existing subscription eval runner.
• Sampling, not full replay, keeps nightly journeys to 25–50 cases.
• Fable stays out of the loop for this — subagents run the checks, not the orchestrator.

Proves: CLAUDE.md "Evals run on the subscription" and "Token budget" rules; prior $600 metered-key incident.

8. Decisions for Fede

Four short yes/no calls, each with a recommendation already picked — nothing here needs him to design anything from scratch.

• Wire the nightly journeys now, or wait until after Thursday's Western Slope go-live? Recommend: after Thursday, canary dark first.
• Add a daily synthetic call/text to the live 970 line? Recommend: yes — one call a day, team-number allowlist only.
• Fix the PR-gate missing key so drift checks stop silently skipping? Recommend: yes, small PR.
• Put the coworker harness on a nightly timer? Recommend: yes.

Proves: each item traces to a named gap in Chapter 3 or the field pattern in Chapter 4 — none is net-new scope.

A1 · Appendix — full record

Every table behind the chapters above, for whoever builds this next.

Every scheduled GitHub Actions workflow

Harnesses and their trigger

Component-pair gaps

Industry sources

Full inventory: ~/agents/006/propflow-testing-inventory-2026-09-13.md. Full industry research with all 24 sources: ~/agents/006/agentic-testing-industry-research-2026-09-13.md.