ADR-0086 — Eval Platform: Morpheus, Sherlock, and the Replication

• Status: Accepted — this ADR ratifies + governs the eval platform. It documents what is built and live as of 2026-07-11 (the Morpheus generate→prove→approve loop and its Agent-Smith bridge) AND fixes the first principles + invariants that all future work on this system must honor. Aspirational pieces are labeled inline as [ASPIRATION]; shipped facts are labeled [LIVE] where the distinction matters.
• Date: 2026-07-11
• Deciders: Gera (owner); Fede; coordinator sessions aeb2d1da (gate scoping, 2026-07-08) + the 2026-07-09→11 generation/bridge build sessions.
• Supersedes / relates:
  • Amends ADR-0064 (Morpheus = eval executor). ADR-0064's charter "runs the real eval" is widened here to "creates AND runs the simulations" (the amendment already lands in ADR-0064's header). This ADR is now the canonical governing reference; ADR-0064 remains the record of the executor design.
  • Relates to ADR-0065 (deep self-discovering evals — the investigator-Sherlock, completeness critic, claim→manifestation graph). ADR-0065 is [ASPIRATION], deferred — 0% built. This ADR governs the platform that exists; ADR-0065 is the deep-evals north star it does not yet reach.
  • Relates to ADR-0061 (Maestro package→queue routing), ADR-0030 (PMS owns WO ids — the "real is real" id rule), ADR-0026 (Temporal PII boundary), ADR-0078 (The Construct = Smith's scheduler, a different system — not reused here).
  • Deeper "why" lives in the planning docs (this ADR makes their vocabulary + invariants canonical; the mechanics stay there): docs/planning/replication-morpheus-gate.md (the authority on names + the create/validate firewall + D1–D9), docs/planning/morpheus-generation-design.md (the generate→prove→approve build + the three Agent-Smith bridge pieces), docs/planning/replication-morpheus-and-smith-open-work.md (the honest open-work map).

Context

PropFlow is an AI-agentic-native platform: Clara handles real tenant maintenance, leasing, renewal, and voice flows against real AppFolio, real Twilio, and prod DynamoDB. An agent is only as trustworthy as our ability to tell when it is lying to us — so the eval platform's job is to prove Clara actually did the thing, on real surfaces, with receipts, and to feed that proof into the loop that ships changes.

The problem this ADR solves is not a missing capability. The hard part — a prod-faithful, real-source-verified, ledger-captured harness with a machine-readable verdict — already exists and is excellent. The problem is that the knowledge is scattered: across ~6 planning docs, two aspirational ADRs (0064/0065), and a large body of hard-won session learnings that live nowhere durable. The vocabulary drifted (the UI credited "Sherlock" for grading that Morpheus's harness actually did), the create-vs-validate boundary was drawn backwards, and the failure modes that bit us building the generation loop exist only in session memory.

This ADR is the single governing reference. Someone six months from now who must modify or repair this system should read only this ADR + its linked sources and understand the whole thing: the first principles, the cast and their firewalled roles, the end-to-end flow, the invariants that must never break, the known failure modes and their fixes, and the operate/debug/extend playbook.

Naming and the two load-bearing firewalls were locked by Gera on 2026-07-08 (replication-morpheus-gate.md §0, decisions D1/D2/D5). This ADR does not re-litigate them.

Decision

1. First principles (the two load-bearing firewalls)

Everything below is downstream of these two. If a change would weaken either, it is wrong by construction.

FIRST PRINCIPLE — the create/validate firewall (D2). The thing that GENERATES tests (Morpheus) must NEVER be the thing that GRADES them (Sherlock). A self-improving eval loop's single biggest risk is that it learns to pass by lowering the bar. Once Morpheus generates scenarios (D1), fusing validation into him creates exactly that conflict of interest. So the platform is one system, two firewalled roles: Morpheus creates + drives; Sherlock validates. The gate consumes Sherlock's verdict, never Morpheus's self-report. (Same rule ADR-0065 §7 states constitutionally; this ADR makes it the platform's #1 invariant.)

SECOND PRINCIPLE — the human ✅ is the oracle guard (D1). Generated scenarios are born approved:false. Only a roster human's reaction stamps approval; Smith never approves its own work (the reviewer name is always the human who reacted — morpheus_author.py approval path). Receipts before review: a generated scenario must prove GREEN on the real bench before a human is asked to approve it. This is what keeps the oracle problem ("a vibe judging a vibe") from reappearing: Morpheus proposes, the bench proves, a human disposes.

2. The cast + their firewalled roles (vocabulary LOCKED — do not re-litigate)

What a golden is. A golden = a predicted conversation script: the hard, predictable checkpoints of a flow (said X, called tool Y, and did not do the wrong thing) with the in-between deliberately fuzzy (many agents, many tool calls). That fuzziness is why it's Morpheus — "not a straight line." Clara's replies are produced live by the real agent; only the tenant's inbound beats are scripted. Morpheus grades whether the fuzzy path hits the predictable hard checkpoints.

Levels L1→L4: L1 single-party (created a WO / "hello") → L2 multi-turn → L3 multi-party (tenant + handyman) → L4 full investigation. Today: L1, text-lane maintenance only.

Morpheus builds and drives the simulation; Sherlock decides whether Clara actually did the thing; the Replication's merge gate consumes Sherlock's verdict. Today this sentence is true for the standalone loop (morpheus-run-one.ts ANDs Sherlock into its exit code — §4) but is [ASPIRATION] for the Replication merge gate: the connector (§4, Piece 3) validates + curates post-merge; it does not yet block a merge on a red. See §7 (open work).

3. Sherlock's two passes (the boundary, drawn correctly)

The confusion Gera flagged was real and was a boundary bug, not a role bug. The validation people credited to "Sherlock" was split across Morpheus's harness and a narrow post-hoc auditor. D2 redraws it: Sherlock = all validation, two passes.

• In-run pass — the checkpoint grading in src/lib/maintenance/eval/checkpoints.ts (2,546 lines): seven manifestation lenses (ddb, s3, appfolio_api, propflow_ui, appfolio_ui, twilio, vendor_conversation) via independent fresh reads, plus clara-judge as a fail-closed LLM wording lens (checkpoints.ts:232 — a null judge result is NOT passed). This is re-labeled as Sherlock's in-run pass; the UI copy already reflects it (src/components/maestro/checkpoint/Sherlock.tsx:9-10, "Sherlock's IN-RUN pass (locked D2)").
• Post-hoc pass — the fake-detector (src/lib/domain/evals/fake-detector/): pure claim extraction → independent fresh re-reads → fail-closed classification verified | fake | unverifiable (detect.ts:10). "Sherlock was a liar, pushed to always tell the truth." Unreadable = unverifiable, never verified. The verdict is clean IFF zero fakes AND zero unverifiable (detect.ts:46-49, 188) — strictly stronger than the old CLI's fake===0.

Deferred: ADR-0065's investigator-Sherlock (strategy toolkit, halt authority, versioned wisdom config, per-gate LLM deputies) is [ASPIRATION], not now. The deterministic receipt-required auditor is sufficient for gate v1 and is trustworthy precisely because it is not an LLM. Until scenarios are generated at variety, there is nothing for a per-gate investigator to discover that the specs don't already enumerate.

4. The end-to-end flow

Two entry paths converge on the same authoring pipeline.

 ENTRY A — manual/curated (Slack)          ENTRY B — the connector (auto, #3587)
 "author scenario [slug]: <hint>"          Replica PR merges + Vercel build green
   in #agent-smith / DM / member chan        → SmithApprovalWorkflow.build_passed seam
   (roster members only)                      → path-map gate (clara-behavior-evals.yml,
        │                                        read at RUNTIME) — no match ⇒ not_applicable
        │                                      → MODEL_LOW hint derivation w/ honest carve-out
        │                                        (text-lane MAINTENANCE only ⇒ else not_coverable)
        └───────────────────┬──────────────────┘
                            ▼
        ┌──────────────────────────────────────────────────────┐
        │  AUTHORING PIPELINE (detached worktree off fresh main) │
        │                                                        │
        │  1. GENERATE  scripts/eval/morpheus-generate.ts        │
        │     Opus-4.8 authors ONE net-new L1 golden from an     │
        │     exemplar + the correctness rubric + HARD RULES 1-13│
        │     → static committed JSON, provenance.approved:false │
        │     (self-validates via validateGeneratedGolden)       │
        │                                                        │
        │  2. PROVE     scripts/eval/morpheus-run-one.ts         │
        │     Lane B: branch code on the mini, in-process        │
        │     Temporal, against the appfolio-45 bench (real      │
        │     Twilio/L4/prod DDB). --report-only for unapproved. │
        │     Bench lock (one-at-a-time). Writes                 │
        │     GOLDEN_EVAL_BRANCH_HEAD.                            │
        │     ── Sherlock in-run (checkpoints.ts → gridAllGreen) │
        │     ── Sherlock post-hoc (auditCapturedRun → clean)    │
        │     exit 0 ⇔ grid ALL GREEN **AND** verdict.clean      │
        │                                                        │
        │  3. PROPOSE   green → Slack proposal ("React ✅/❌")   │
        │     + maestro:<runId> pointer + Maestro-tab link       │
        │                                                        │
        │  4. HUMAN ✅  a roster human reacts → morpheus-approve │
        │     stamps approved:true + reviewedBy=the human        │
        │     → git add <artifact ONLY> → commit → push → PR     │
        └──────────────────────────────────────────────────────┘
                            ▼
        MERGED ARTIFACT joins the NIGHTLY RERUN (05:00 CT,
        co.propflow.morpheus-nightly-daily) — quality-regression watch,
        one-at-a-time, provenance.approved===true only.

Where each named component sits:

5. The two lanes (why Lane B is the gate lane)

Lane B is the gate lane because only an in-process run from the PR's checkout exercises the PR's worker code. The honest nuance on "no staging": there is one AppFolio account (jpco), so the isolation boundary is the bench (appfolio-45 + fake personas), not the environment (ADR-0065 §9). Validation runs against prod services (prod DDB, deployed L4, real Twilio) but Clara's reasoning runs from the mini's main checkout — the same code as deployed.

6. The invariants (a future modifier must not break these)

1. The D2 firewall. Morpheus never grades its own output. A generated scenario is DATA (turns + a checkpoint spec — it declares what to check, never how a lens reads). Grading code (checkpoints.ts / clara-judge / the fake-detector) reads only fixed source.
2. The human-approval oracle. provenance.approved:false by default; only a roster human stamps approved:true; Smith never approves its own work. The nightly discovers provenance.approved === true artifacts by strict identity (morpheus_nightly.py:182).
3. Lane-B-only for gate runs. Branch code on the mini vs the prod-fleet preset lane. Vercel cannot host an in-process Temporal dev server; the golden runner refuses --channel voice and refuses voice-PRIMARY goldens by name (run-golden-eval.ts:13, 420-421).
4. appfolio-45 blast-radius firewall. EVAL_PROPERTY_ID = 'appfolio-45' is pinned; the drive/inject firewall throws on any other property (run-golden-eval.ts:124, 237, 477). Both *-eval queue overrides are REQUIRED — the runner refuses prod queues (run-golden-eval.ts:32).
5. approved:false default + MORPHEUS_ALLOW_UNAPPROVED=1 only on the report-only lane. The registry refuses an unapproved artifact as a real gate; only --report-only (which sets the env) lets it run for observation (registry.ts:112-114, morpheus-run-one.ts:79).
6. Static-DATA artifact — no LLM at run time. The LLM runs once, at generation time; the runnable artifact is static committed JSON, so run-to-run determinism is exactly the exemplar's (Clara's own variance absorbed by grace:'fuzzy' + clara-judge + retry-once).
7. One-at-a-time bench. The bench lock (scripts/eval/bench-lock.ts) serializes the PR gate, the nightly, and manual runs; morpheus-run-one.ts exits 3 when the lock is held. precleanBench would wipe a concurrent run mid-grade — this is why the lock exists.
8. Metered ANTHROPIC_API_KEY for generation + run. Clara's agent loop and generation must ride the metered key, never the subscription bearer's shared weekly cap — else a run 429s (morpheus_nightly.py:41-42, fail-loud when missing).
9. No real phones / no PII, ever. Persona phones are the compiler tokens {{TENANT_PHONE}}/{{HANDYMAN_PHONE}} (never a real E.164, in personas OR message text — these messages are really SENT). The generator hard rule 1 + validateGeneratedGolden check (a) enforce it; the connector's hint derivation fails loud on any phone-like token.
10. The generator's HARD RULES are the encoded contract. morpheus-generate.ts:311-326 (rules 1–13). The validator (golden-schema.ts) mechanically enforces the subset that is checkable; the rest bounce at human review.

7. What is LIVE vs what is NOT (honest current state + coverage boundary)

LIVE [2026-07-11]:

• Generation (morpheus-generate.ts), the prove harness (morpheus-run-one.ts) with Sherlock ANDed into the exit code, pre-run reset, the play/pause stepper (ADR-0064 executor), the nightly rerun, and the full Slack authoring loop — proven end-to-end 2026-07-11 (first L1 generated golden green on prod with Sherlock validating; gen-i3-pivot-ambiguity-bathroom-mirror-cracked authored by Smith + approved by Gera, #3586).
• The connector (#3587) is merged but effectively unfired — it validates + curates post-merge; see below.

NOT live / coverage boundary — state precisely, no overclaiming:

• Only text-lane MAINTENANCE tenant flows are authorable. Voice / leasing / renewal / turnover → the connector returns not_coverable honestly (morpheus_validation.py:29-34). The SMS golden runner is the only authoring lane; voice-primary goldens are refused by the runner.
• The connector VALIDATES + curates; it does NOT BLOCK. It fires on build_passed (merged + deployed), is patch-gated + fail-soft (can never break the approval reply), and lands its receipt in the task's Slack thread. It is not a blocking pre-merge gate. A Replica PR that rewires the maintenance loop merges today on unit tests + a bot review; the golden gate is post-merge/advisory.
• Maestro is built but UNWIRED into prod inbound (ADR-0061 pipeline exists + deployed; prod inbound still uses the pre-Maestro dispatcher; exercised only in the eval lane).
• No staging bench (validation runs against prod services post-deploy — see §5's honest nuance).
• L2–L4 golden levels are unbuilt. So is the perturbation generator (P3a) and exemplar-guided flow-suite generation (P3b).

Consequences

Known failure modes + fixes (the debug playbook)

These bit us building the generation loop; they exist nowhere else durable. Symptom → root cause → fix/where.

Standing gotchas (not one-off):

• SR-id vs WorkOrderId drift. createWorkOrderL4 returns the SR id; delete/cancel need the WO id — off by one. Resolve the WO id before any terminal op.
• Slack MCP artifact. The Slack MCP appends "Sent using Claude" to messages; the parser swallows it into the issue-hint. Harness-only artifact — strip it, don't treat it as content.

Operate / extend playbook

Run one scenario manually (the Lane-B env contract). The proven shell contract (ported into morpheus_nightly.py:38-42):

source .env.local (OVERRIDING the process env)
unset TEMPORAL_API_KEY TEMPORAL_ADDRESS TEMPORAL_NAMESPACE   # force the local dev server
export DYNAMODB_TABLE_NAME=propflow-prod                     # .env.local defaults to stage
export APPFOLIO_EVAL_ACCOUNT_ID=jpco
# the metered ANTHROPIC_API_KEY MUST be set (fail-loud) — Clara must not ride the subscription cap
npx tsx scripts/eval/morpheus-run-one.ts --scenario <id> [--report-only] [--timeout-sec 1200]
# exit 0 ⇔ grid ALL GREEN AND Sherlock clean; 1 = fail; 2 = usage; 3 = bench-lock held

Operate the nightly. Runs as step 6 (last) of NightlyQueueWorkflow, the 02:00 CT Temporal batch (folded in 2026-07-12; the standalone 05:00 LaunchAgent co.propflow.morpheus-nightly-daily + its installer were retired). Ordered last so its appfolio-45 bench use never overlaps the earlier maintenance-eval + turnover-eval sub-jobs (the queue is strictly sequential); its per-job timeout is raised to 75 min. Still records its OWN AutomationRun under co.propflow.morpheus-nightly-daily (morpheus_nightly.py JOB_ID). On-demand: cd tools/agent-smith && uv run python -m agent_smith.morpheus_nightly, or fire the whole queue via uv run python -m scripts.register_nightly_queue_schedule --trigger. Local-ops cutover on the mini: launchctl unload ~/Library/LaunchAgents/co.propflow.morpheus-nightly-daily.plist + delete that plist so the old 05:00 job stops firing.

Deploy Smith-side changes. After merging any change to the nightly / authoring loop / connector, deploy with ~/.claude/scripts/smith-sync-from-main.sh (restarts the Temporal worker + Slack socket; the socket carries the trigger/reaction wiring).

The discovery contract. The nightly runs only provenance.approved === true artifacts (strict identity — an unapproved artifact never gate-runs; every skip is logged with its reason).

Extend coverage to a new flow lane. Today the honest carve-out gates it: the authoring lane proves ONLY text-lane MAINTENANCE tenant flows. To add voice / leasing / renewal / turnover you must (a) give that lane a runner that Lane B can drive (voice is EL-simulate, not the golden runner — which refuses voice-primary), (b) teach the golden schema that lane's anchor type (message-beat | event | programmatic — checkpoints must land at the right moments: message-turn for SMS, event/webhook/cron for voice), (c) author that domain's exemplar goldens, and (d) update the connector's coverage carve-out prompt so it stops returning not_coverable for that surface.

Open decisions (carry forward — do NOT resolve here)

From the gate doc §6; only Gera can call these:

• D3 — Tier-2 gate host. Mini as phase-1 (single point of failure; holds ~15 secrets + branch worktrees) vs hold for a remote executor vs mini-now + written migration path.
• D4 — Path-map home. GitHub paths: filter only, or also a versioned config read by merge_gate.py as a 4th gate leg? Which carries the drift-guard test?
• D6 — Gate target. Wire Maestro into prod first (who drives PR1?), or gate the current inbound path?
• D7 — Cost/latency budget per gated PR. Full single-golden run (~5–10 min est., unmeasured; real SMS + L4 writes; serialized bench) per Replica PR vs promptfoo-only per PR + goldens nightly. Concurrent Replica PRs mutex on the single bench — acceptable?
• D8 — Voice carve-out. Promote EL-simulate to a first-class gate lane for voice-surface Replica PRs (cheap, no bench writes, parallel-safe, trajectory-asserting)? Reserve the PSTN robot tier for pre-arm sign-off.
• D9 — Flake / variant-failure policy. Hard-fail-cells + one retry (recommended) or stricter? Is report-only → promote/quarantine the right P3 bar? Who owns quarantined variants; who's paged on a template-golden red?

Plus the unbuilt (labeled, not decisions): L2–L4 golden levels; the perturbation generator (P3a) + exemplar-guided flow-suite generation (P3b); and — most importantly — the Replica-PR-blocks-merge-on-red gate. Today the connector is post-merge/advisory: it VALIDATES + curates, it does not yet BLOCK. Turning it into a blocking pre-merge gate is future work gated on D3/D4/D6/D7.

What this ADR makes canonical vs what stays in planning

• Canonical here (govern from this doc): the cast + firewalled roles, the two first-principles, the invariant list, the LIVE-vs-not coverage boundary, the failure-mode debug table, and the operate/extend playbook.
• Stays in the planning docs (the deeper "why" + mechanics): the phased build plan P0–P4 and T-shirt sizes (replication-morpheus-gate.md §5), the generation build internals + the three Agent-Smith bridge pieces (morpheus-generation-design.md), and the consolidated open-work map (replication-morpheus-and-smith-open-work.md). When those docs and this ADR disagree on a principle or invariant, this ADR wins; on mechanics, the planning doc is the live detail.

Alternatives considered

• Fuse validation into Morpheus (drop Sherlock). Gera was open to it; deferred to this design. Rejected: it destroys the D2 firewall exactly when D1 makes Morpheus a generator — the generator would grade its own tests and could learn to pass by lowering the bar. Kept as one system, two firewalled roles instead.
• Gate on the golden run inside GitHub CI. Rejected: would require exporting ~15 secrets (prod-write AWS, L4 token, Google OAuth) to GitHub, cannot serialize against the mini's bench runs, and — decisively — a GH runner cannot execute the PR's in-process Temporal-worker code. The mini already holds every credential and builds Replica worktrees. (ADR-0064 already rejected Vercel/GH-Actions/ECS as the executor host for the same class of reasons.)
• Perturbation-only generation (vary a frozen golden). Kept as the P3a stepping stone, not the end state: a perturbation over a frozen spine can vary how a scenario is expressed but cannot discover a variant that should produce a different correct outcome. The target (P3b) is exemplar-guided full-spec generation with human review — the oracle-problem answer.
• A LLM at run time (dynamic scenario). Rejected for the runnable artifact: it reintroduces run-to-run nondeterminism and "a vibe judging a vibe." The LLM runs once, at generation time; the artifact is static committed DATA.
• Block Replica merges on the golden gate now. Deferred: the connector ships post-merge/advisory because the blocking-gate host (D3), path-map home (D4), gate target (D6), and cost budget (D7) are unresolved. Advisory-first lets the platform earn trust (green receipts in the task thread) before it can veto a merge.