ADR-0065 — Deep, self-discovering evals: the claim→manifestation graph, Sherlock as the receipt-required gatekeeper, and the fail→fix→teach→retry loop

• Status: Proposed (scoped 2026-06-20)
• Deciders: Gera (+ driver session 7e2bcadd)
• Relates / builds on: ADR-0064 (Morpheus — the eval executor), ADR-0061 (maestro package→queue), #2528 (Sherlock — the fake-detector), #2524 (Evidence Ledger), #2530 (Wall of Shame). Extends the golden harness (scripts/temporal/run-golden-eval.ts → runGoldenScenarioGrid) and the read-verified discipline.
• Cast (user-facing names only; Temporal stays domain-named): Maestro routes inbound → Morpheus runs the eval for real → Sherlock investigates & gates each checkpoint → Wall of Shame posts the verdict → Evidence Ledger keeps the record. Above them: Claude (the driver/runner that authors + fixes the evals) and Gera (above Claude).

Context

Our agent evals work — but they gloss. A checkpoint typically verifies the local, easy thing (a DDB row exists → green) and never checks the full manifestation of what the agent claimed. "Tenant photo added to the work order" should be true in ~five places; checking one is the gloss. The concrete pain that triggered this ADR: work-order images stopped rendering at the WO level on both PropFlow and AppFolio, and the eval did not catch it — the photo row was in DDB, so it went green, while the image was never actually on either UI.

The unacceptable fix is "tell the eval to check images." If a human has to enumerate what to look for, the eval is shallow. The eval must discover the gap itself — "a photo was claimed; did it actually manifest everywhere it should, including the UI? No? Fail." — without being pre-told.

This ADR defines how evals go deep, stay honest, and self-improve, so we can trust them enough to scale the pattern from maintenance to leasing, tours, and renewals.

Why harnesses gloss (the root cause)

They verify one cheap surface and call it done. Depth requires two things they skip:

1. Verify every layer a claim must touch (backend + cross-system + frontend), not the nearest one.
2. Derive the layers from the claim rather than from a human's static checklist.

Decision

1. "Done" is defined by a claim → manifestation graph

Every action the agent takes has a manifestation set — the places it MUST become true. Example, "tenant photo added to WO":

A checkpoint is green only when every manifestation is independently confirmed at the layer it lives. The image bug is caught because the graph for "photo" includes (4) and (5) — the eval goes red there on its own.

The manifestation graph has two halves so it's neither brittle nor human-prescribed:

• Deterministic backbone — a map from agent tool-calls → expected manifestations. Mechanical, auditable, the spine. Lives in the per-domain wisdom config (§5).
• A completeness critic (the "consciousness") — an LLM pass over the agent's full action log + everything verified so far, asking: "what did the agent claim that we have NOT independently confirmed at every layer it should appear — including the UI?" It generates new checks, which then run. This is the self-discovery: unprompted, it says "a photo was added but nobody confirmed it renders on PropFlow or exists on AppFolio" and goes to find out.

2. The frontend is a first-class verification layer (two sources of truth)

Backend read-validation proves "the data is in DDB/AppFolio." The frontend check proves "the user actually SEES it" — catching the "we think we uploaded the image but didn't" / silent-render-failure class. Truth = backend-read AND frontend-render both confirm.

• The HTML/DOM is what's validated; the image is for human viewing. We do not pixel-guess. The capture returns the rendered HTML alongside the image, and we assert against the DOM (is there an <img> for this photo? does the status node say "Completed"? is the rating 5?). The picture stays so a human can glance; the HTML is the machine proof.
• Full-page capture, not just the top viewport (below-the-fold content — the uploaded image, the activity log — must be in the shot).
• PropFlow — we own the DOM → deterministic. Instrument components to self-report eval-state via data-eval-anchor / a small data-eval JSON blob ("tap into the variable"). DOM-assert + element-crop the exact thing that changed. No LLM needed → cheap. Start with Work Orders.
• AppFolio — we don't own the DOM → lean on the verifier LLM + their IDs. We can't add anchors; we point a Haiku at the page guided by AppFolio's WO/field IDs ("confirm the Status field = X; confirm an attachment is present"), with full-page capture. Gated on the AppFolio browser-session refresh (the hybrid-stale OIDC bounce — see the appfolio recon notes). Do not over-engineer screenshot intelligence here.

3. The per-checkpoint verifier — receipt-required, "actually done vs vibe"

Between checkpoints, an LLM pass (Haiku default, escalate to Sonnet for ambiguity / AppFolio / partial renders) asks exactly three questions:

1. Goal — what was this beat supposed to achieve?
2. Schema — what does "done" concretely look like (the manifestation set)?
3. Did we actually do it, or was it a vibe? — judged against the real evidence gathered, per layer.

The one rule that keeps it from being a vibe judging a vibe: the verifier earns a "done" verdict only when it can point at the real read/render that proves it — a DDB read, an AppFolio read, the PropFlow DOM showing the <img>, the AppFolio page showing the attachment. No receipt → not confirmed. Same default-skeptic invariant Sherlock already enforces (verified ⇒ receipt), now applied to the reviewer itself.

It has two jobs, both grounded in reads: judge each objective against the gathered evidence with the receipt, and critique completeness (ask for the missing read/render, then re-judge). This deepens the existing per-checkpoint clara:judge lens (no fork): its input grows from "did the message look right" to goal + schema + multi-layer real evidence (incl. the frontend render), plus the completeness-critic + receipt requirement.

4. Sherlock — the investigator stationed at every gate (border patrol)

Sherlock is the agent living at the checkpoints: curious by default, skeptical by mandate, devil's advocate. He does not trust the claim — he investigates it.

• Strategy toolkit. Like a real investigator, Sherlock has a kit of confirmation strategies ("to confirm images: try get_work_order_images; read the WO page DOM; read the AppFolio attachment list"). He picks strategies per checkpoint, runs them, collects receipts, judges.
• Halt authority. Sherlock can break the flow — "you shall not pass" — and fail a gate with structured context. When a gate is broken there's no point continuing.
• Receipt-required. He approves only what he could independently confirm. No receipt → no pass.
• Per-domain wisdom config (§5) tells him the checkpoints, the manifestation schema, and the strategy registry for that workflow.

Sherlock's failure is machine-actionable — not a vague "it failed":

{ gate, claim, strategiesTried[], whatWasMissing, suggestedCapability }

e.g. { gate: "complete.closed-with-proof", claim: "completion photo uploaded", strategiesTried: ["ddb-read OK", "propflow-DOM: no <img>"], whatWasMissing: "no get_work_order_images method to confirm AppFolio attachment", suggestedCapability: "get_work_order_images(woId)" }.

5. The per-domain wisdom config (the replication unit)

Sherlock's knowledge for a workflow lives in a versioned config — e.g. sherlock-maintenance.checkpoints.json — carrying: the ordered checkpoints, each checkpoint's manifestation schema (the claim→layers map), and the strategy registry (how to confirm each manifestation). Morpheus has its matching per-domain scenario. The replication unit is { Sherlock domain-wisdom + Morpheus scenario + manifestation schema }. Harden maintenance first; standing up leasing / tours / renewals = author that domain's wisdom + scenario and the same machine carries it. This is how "renewals → tours → leases" becomes tractable instead of hand-written each time.

6. The fail → fix → teach → retry loop (self-improving)

The closed loop that turns a failed gate into a permanent improvement:

1. Morpheus runs the scenario; Sherlock investigates each gate.
2. Gate fails (e.g., can't verify images — no method). Sherlock halts; Morpheus stops the run and relays the structured failure up to the driver (Claude).
3. Claude (driver) investigates → fixes the real gap (usually: builds the missing verification capability, e.g. a get_work_order_images method) → PR → merge.
4. Claude teaches Sherlock — the fix is not a code comment; it persists into Sherlock's wisdom config (a new strategy: "to confirm images, use get_work_order_images"). Next run he's smarter because the config changed.
5. Fresh retry (clean restart, no carried-over state that could mask the gap) → Sherlock uses the new strategy → gate passes → next gate.

Each cycle improves both the system (gains the missing method) and Sherlock (gains the strategy). End state: every gate green, every image + every context confirmed — confidence.

7. Anti-gaming guard (constitutional, non-negotiable)

The biggest risk of a self-improving eval loop is that it learns to pass by lowering the bar. Absolute rule: a fix must make the real thing verifiable; it may NEVER weaken or bypass a gate to go green. Sherlock stays adversarial. "Teaching Sherlock" only ever adds a way to confirm reality, never a way to assume it. This is the same honesty rule that is already the platform's spine (verified ⇒ receipt, no mock theater, read-the-real-source).

8. Checks & balances — who watches the watchers

The watchers (verifiers, Sherlock, Morpheus) can themselves be wrong. Four mechanisms, cheapest first:

1. Receipt-grounding (mechanical). A "pass" with a missing/weak receipt is detectable without another LLM — the first line against silent false-passes.
2. Sampled adversarial second opinion. On high-stakes gates (or a random audit sample), a different model/lens re-judges; disagreement → escalate. Not every gate (cost).
3. A complaint inbound to Claude (the apex). Every run emits not just pass/fail but signals from each actor — Sherlock: "missing capability X"; a verifier: "ambiguous, low confidence"; Morpheus: "couldn't drive gate 3 / malformed package." Claude ingests the batch → a triage list → fixes the system, the scenario, or the bad watcher itself (rewrite its prompt/wisdom). A Haiku doing a bad job gets flagged and improved by the same loop.
4. Transparent logging = spot-audit. Every verdict + receipt lands in the Evidence Ledger / Wall of Shame, so a human (Gera) can review "did Sherlock approve something he shouldn't have?" The transparency is a check.

The hierarchy: agent → Sherlock judges it → receipts + cross-checks + complaints judge Sherlock → Claude is the apex court (can rewrite any watcher) → Gera is above Claude.

9. Dev-cycle: bench-on-prod + a local fast loop (not a staging detour)

AppFolio is one real account (jpco) — there is no "staging AppFolio." The isolation boundary is therefore the bench (the appfolio-45 test property + fake personas), not the environment. So:

• The appfolio-45 bench (on prod) IS the eval surface — safe to write to by construction; prod is the end goal anyway.
• Fast loop = local code → prod-bench data → view locally / on a phone over LAN. No deploy, no preview, no "viewer loop." (Local dev requireAdmin is open, so the eval page is reachable on the LAN without auth.)
• A staging-writes-to-AppFolio flag is not the answer — it would point stage-DDB at the same real AppFolio bench (no isolation gain, more complexity). If a staging path is ever wanted, the honest version is "seed the bench + eval-capture infra on stage," not a flag.
• Reserve an actual prod deploy for the final "does it work deployed" check.

10. Cast, roles & visibility — one trace, roles as lanes

The cast (Maestro/Morpheus/Sherlock/verifiers/Claude) is confusing today because the UI is tabs-per-feature (a Maestro tab, an Evidence Ledger tab, a Wall of Shame tab) — which reads as "what are these and how do they relate?" The fix is not an interface per agent. It's one shared eval-session trace (one runId, one timeline) where each role is rendered where its job physically is, so the relationships are legible by construction.

Decision: a single run trace; roles are lanes within it; the Ledger and Wall of Shame are views of that run, not separate things.

Lay the run out as a checkpoint timeline (gates left→right) and place each actor at its job:

• Morpheus = the driver → the play-head moving across the timeline. You watch him advance the scenario checkpoint by checkpoint. "Morpheus runs the eval" is self-evident because he is the motion (play-all / play-to-checkpoint / pause / step).
• Sherlock = the gatekeeper → a pass/fail stamp on each gate (✓/✗ + receipts). He's stationed on the checkpoints. "Sherlock decides who passes" is self-evident because he is the gate.
• The verifiers (Haiku/Sonnet) = Sherlock's deputies → they nest UNDER a gate when expanded (strategies tried, receipts, screenshots/HTML). Not top-level actors — they're how Sherlock investigates.
• Maestro = the subject under test (not a watcher) → its decisions are the thing being judged inside each checkpoint, not a peer of Sherlock.
• Claude (driver) = apex, above the run → a cross-run triage / complaint inbox view (what failed across runs, what needs a fix), because it spans runs — not inside a single timeline.
• Wall of Shame / Evidence Ledger = the record → the verdict facet + the deep-evidence facet of the run, reached from it.

The relationship then reads straight off the screen — a pipeline you can see: Maestro is judged → Morpheus drives → Sherlock gates → Claude fixes → Gera oversees.

Two cheap legibility aids (required):

1. A persistent cast legend on the eval page — one line each, always visible: Morpheus — runs the eval · Sherlock — investigates & gates each checkpoint · Maestro — the brain under test · Claude — fixes what fails · Ledger/Shame — the record.
2. Role-consistent identity — one icon + color per actor, used on every surface. Wherever an actor's color appears, its role is known; no re-learning per screen.

Worked example — the missing-images bug (the design must catch this on its own)

1. Agent adds a tenant photo + completes the WO. The action log shows request_photo → photo attached → WO mirrored → completed.
2. The completeness critic derives, from the action log, that "photo" must manifest at DDB ∧ S3 ∧ AppFolio ∧ PropFlow-UI ∧ AppFolio-UI — and notices the last two were never gathered.
3. It asks Sherlock to confirm the UI renders. Sherlock reads the PropFlow /maintenance/<id> DOM → no <img> for the photo → and has no strategy to read the AppFolio attachment → FAIL, with {whatWasMissing, suggestedCapability: get_work_order_images}.
4. Morpheus halts, relays to Claude. Claude finds the real render bug, fixes it, adds get_work_order_images, teaches Sherlock (new strategy in sherlock-maintenance.checkpoints.json), retries fresh.
5. Next run: Sherlock confirms the <img> renders + the AppFolio attachment exists → gate passes. The bug can never silently recur (the manifestation is now permanently in the schema).

No human ever said "check images."

Consequences

Good

• Evals that actually justify confidence — every claim confirmed at every layer, frontend included.
• Self-discovering: catches gaps nobody pre-listed (the whole point).
• Self-improving: each failure permanently hardens both the system and Sherlock.
• A clean replication unit to scale across domains.
• Honest by construction (receipt-required end-to-end, anti-gaming guard, watch-the-watchers).

Costs / risks

• LLM cost per checkpoint (Gera: "expensive but worth it") — mitigated by Haiku-default / Sonnet-escalation + sampled cross-checks.
• Loop must be bounded (max retries; halt-vs-continue policy) so a stuck gap doesn't spin.
• The completeness critic could hallucinate checks — bounded by the deterministic backbone + the receipt requirement (a hallucinated check that can't be grounded in a real read is itself discarded).
• AppFolio frontend verification is inherently softer (no DOM ownership) — accepted; lean on the LLM + IDs, gate on the session refresh.

Open questions (resolve before/while building)

1. Loop bounding: max retries per gate; when does Morpheus halt-and-continue vs halt-and-stop; how to detect "the same gate keeps failing the same way" (no progress).
2. How "teach Sherlock" writes to the wisdom config — schema of the strategy registry; who/what authors the entry (Claude proposes → human-reviewed? auto-merged behind the anti-gaming guard?).
3. The complaint-inbound format + transport — how Sherlock/Morpheus/verifiers emit signals and how Claude ingests them (a per-run eval_experience artifact? a DDB row? a structured report the driver reads?).
4. Cost ceiling — per-run LLM budget; which gates get the sampled second opinion.
5. Capture-by-default placement — moving capture from the golden script into the harness so every eval entry point gets it (the cookie-mint boundary into the bundled activity needs solving).
6. Where the per-checkpoint verifier physically runs relative to Morpheus (in-process vs an activity) and how it stays read-grounded.

What exists today vs to-build (anchored to reality)

Exists: Morpheus executor (ADR-0064), Sherlock fake-detector (#2528, real-source reads + receipts + trust tiers), Wall of Shame (#2530), Evidence Ledger (#2524), the golden harness + per-checkpoint clara:judge lens, and page screenshots in the ledger (PR #2549 — full lifecycle PropFlow WO-page capture, proven on prod run leak-204-2026-06-20T22:09:46.003Z).

To build (this ADR): the claim→manifestation graph + completeness critic; the receipt-required per-checkpoint verifier (deepening clara:judge); PropFlow data-eval self-reporting + DOM-assert; full-page + HTML capture; Sherlock's strategy toolkit + per-domain wisdom config + halt authority + structured failures; the fail→fix→teach→retry loop + complaint inbound to Claude; the anti-gaming guard; the per-domain replication unit.