PropFlow.ai PropFlow.ai Docs
Proposed (scoped 2026-06-20)markdown source

ADR-0064 — The eval executor runs on Temporal (orchestrator workflow drives signal-stepped execution of the real maestro workflows)

Amendment (2026-07-09 — D1/D2 locked, docs/planning/replication-morpheus-gate.md §0): Morpheus's charter expands from executor-only to scenario supply + drive: Morpheus creates AND runs the simulations. He owns the test corpus — the hand-authored goldens become few-shot exemplars that teach the generator what a good scenario/spec looks like, not merely the fixed tests — plus the net-new generator (perturbation as the stepping stone; exemplar-guided flow-suite generation as the target; every generated spec passes human review before it is gate-eligible). The executor described below is unchanged and remains Morpheus's "drive" half. Per the same decision set (D2), Sherlock = ALL validation: the in-run checkpoint grading is re-labeled as Sherlock's in-run pass alongside his post-hoc fake-detector — firewalled from Morpheus, because the thing that generates tests must never be the thing that grades them. Where the "Name" line above reads "runs the real eval," read "creates AND runs the simulations." The build loop that consumes these verdicts is The Replication (workers: Replicas).

Context

The maestro/maintenance golden eval ("does Clara behave?") today runs only as a local CLI harness (scripts/temporal/run-golden-eval.ts): it boots a local temporal dev server + workers, drives the leak-204 scenario turn-by-turn against isolated *-eval queues + prod DDB, grades each checkpoint, and captures the run to EVAL_RUN#maestro:<runId> (which the /admin/dev/evals Maestro tab now renders as an evidence ledger).

We want to run/test the evals FROM the page like a debugger: a list of evals (each described high-level — "work-order issue → back-and-forth → the Clara→maestro packages it expects"), pick one, press play, watch it execute the whole flow live — play-all or play-to-checkpoint with pause/step — clicking entity tabs (Work Order → AppFolio link + PropFlow link) as they're created, with the maestro diagram showing where the flow is. And it must be read-verified: everything shown is an independent READ of the real surface we claim to have written (read the Twilio Messages API for the SMS; read AppFolio for the WO; read DDB for the row) — never "print what we just wrote."

The load-bearing question: where does the run execute? It can't be the user's laptop forever (not shareable, not "anywhere"), and it can't be a Vercel serverless function (short-lived, stateless — can't boot temporal/workers or hold a paused/stepped session).

Prime directive — honesty over correctness (the Wall of Shame)

PropFlow is an AI-agentic-native platform; our agents are only as trustworthy as our ability to tell when they're lying to us. So the eval playground's prime directive is honesty over correctness: failing an eval is fine — lying about it is not, because a lie hides real breakage and tricks us into thinking something's OK when it isn't. The eval space must be as real as it can possibly get (real phone numbers, real AppFolio, real DDB, real visibility), and any fakeness is caught and recorded.

Definition of truthful (the bar)

A claim is truthful iff it is backed by an independent READ of the real surface it claims to have affected. "We sent a message" is true ONLY if a Twilio read confirms the message exists. The UI showing tenant→handyman "how's it going" is honest ONLY if it reflects a Twilio / AppFolio / DDB read, not the value we just wrote/printed. Read-verified = truth; an unbacked claim is a fake. We read the surface we claim to write — never echo the write.

The Wall of Shame (a first-class, data-backed honesty ledger)

Every detected act of fakeness is forced to a durable, append-only schema and surfaced on the eval page as a growing shame leaderboard. The point isn't punishment theater — it's that we literally have data points for agent dishonesty, trended over time, so the lesson lands: the goal is not to pass evals; the goal is to never lie.

This pillar binds the rest of the design: the read-watchers (below) are both the truth source AND the fakeness detector.

Decision

Run the eval executor on Temporal, as a two-layer workflow design on the existing worker fleet (Temporal Cloud + Fargate), with the Vercel page as a thin client — built directly on Temporal (no separate local MVP step).

Layer 1 — the eval-orchestrator workflow (the "debugger session")

One evalExecutorWorkflow per run (workflowId eval-<scenarioId>-<runId>), on a dedicated eval-orchestrator queue. It:

Layer 2 — the real execution it kicks off

The orchestrator starts the actual maestro/maintenance Temporal workflows — the inbound-package → reasoning-brain path (ADR-0061), maintenanceWorkflow, the comms workflow — on the isolated *-eval queues (INBOUND_PACKAGE_TASK_QUEUE_OVERRIDE, MAINTENANCE_TASK_QUEUE_OVERRIDE, MAINTENANCE_COMMS_TASK_QUEUE_OVERRIDE). This is the real execution, not a simulation — the same code prod runs, physically isolated from real customers' queues.

Read-based verification (a hard invariant)

Every value the page shows is produced by an independent read of the real surface, with a provenance badge (the evidence-ledger contract, extended live): twilio-read (Twilio Messages API), appfolio-read (L4 read / audit), ddb-read (getItem), judge. A "watcher" activity reads the real surface after each action to confirm it landed — we read where we claim to write, never echo the write. Depicted ⊆ verified.

Where each piece runs

The truth pipeline — always-emit → listen → read-verify

The UI must NEVER "assume it wrote, so proceed." It reflects only what the real surface confirms. The chain, end to end:

  1. Always-emit (no eval/prod flag). The real maestro/maintenance workflows emit a domain event whenever they trigger or complete an action (message.sent, wo.created, wo.dispatched, photo.attached, maestro.decision, …) onto an event bus — unconditionally, in every environment. Decision: no isEval flag gates emission. Rationale (Gera): none of this is time-critical, always-on capture is simpler than flag-branching, and a permanent truth feed is strictly more useful. The emit is cheap (an event publish); "it's OK if it's a little slow" — correctness/honesty beats latency here.
  2. Listen (the "got it", not the "assume"). The eval session is a listener on that bus (the webhook-listener model Gera described): when an event lands, it knows a thing happened and when — the trigger to go verify. It does not infer success from "we called the function."
  3. Read-verify (the truth shown). On each caught event, a read-watcher does an independent READ of the real surface — Twilio Messages API for the SMS, AppFolio for the WO, DDB for the row — and THAT read is what the UI shows, with its provenance badge. Emit says when; read says what's real.
  4. Honesty gate. If an emit claims an action but the read can't confirm it (or nothing emitted for a claimed action) → a Wall-of-Shame row. The read is the judge.

Controls — pause / play, no rewind

The orchestrator workflow gives play-all and play-to-checkpoint, with pause and play/step to advance. No rewind (a run is a real forward execution against real surfaces — you can't un-send a text; replay a past run from the ledger instead). On pause, everything shown is linked to the real place: a created WO links to its work-order page (/maintenance/<id>)

Build directly on Temporal (no MVP step)

Per Gera: skip the local-first MVP — build the Temporal implementation directly. The page talks to the run via one controller contract — start(scenario, mode) / step() / resume() / pause() / subscribe() — implemented over Temporal: start = start evalExecutorWorkflow; step/resume/ pause = signals; live state = the event-bus stream the listener accumulates (+ workflow queries for a consistent snapshot). The catalog, read-watchers, CheckpointGrid/MaestroGraph render, and ledger capture are built once against this contract.

Alternatives considered

Consequences

Resolved design decisions (the former open questions)

Still genuinely open (call these out, don't pretend they're settled)

Scope of the first ADR-aligned build

MVP (local-first) lands the UX + the seam now. The Temporal layer (this ADR's core) is the follow-up that swaps the seam — tracked as the next slice. leak-204 is the first scenario; the catalog generalizes after.