PropFlow.ai PropFlow.ai Docs
Superseded by [ADR-0061](./0061-maestro-package-queue-routing.md) — the 3-rung gauntlet framing is replaced by the package→reasoning→action-queue model. ADR-0061 carries forward this ADR's good bones (an LLM proposes a closed vocabulary / a deterministic layer disposes, and the I3–I6 conversational-understanding invariants as requirements of the reasoning brain). Read 0061 for the current model; this ADR is retained as decision history.markdown source

0059 — The maestro is a hybrid command-ladder: an LLM proposes a closed command vocabulary, a deterministic ladder disposes

Context

The problem this ADR closes

The maestro is the orchestrator that decides, per inbound message, which lane it belongs to, which work order it pertains to, what action to take, and what to say back. The contract doc named the failure modes as eight invariants and was honest that only the precedence/ordering class is solved. The conversational-understanding class — I3 ambiguity, I4 multi-context disambiguation, I5 emergent severity, I6 resume-validity, plus the fuzzy detection quality behind I1/I7 — was left OPEN with no proven mechanism. The contract also flagged one load-bearing decision it did not make:

The state-machine-vs-agent-loop fork. Should the maestro be (a) an explicit state machine / dialog stack (deterministic, safe, inspectable — our ladder), (b) a single LLM agent loop with tools + durable memory that drives routing itself (flexible, natural, but an LLM on a privileged path), or (c) a hybrid (LLM proposes, deterministic ladder disposes)?

This ADR resolves that fork and, for each OPEN invariant, proposes a mechanism grounded in prior art with an eval that proves it — or states plainly that it stays open with the closest known approach. It does not claim to have solved conversation routing; it claims to settle the architecture that hosts the judgment, and to make the remaining judgment buildable and measurable.

The standard (locked in from the handoff)

A named invariant is not a solved one. A guardrail with no proven mechanism and no failing-then-passing eval scenario is a wish, not a guarantee. Every mechanism below ships with (a) the prior art it is based on, (b) a concrete buildable shape, and (c) an eval scenario written to fail on today's behavior plus the metric that proves it. Where a requirement has no buildable mechanism yet, it is marked OPEN with the closest approach — not dressed up as solved.

What prior art actually says (the research that grounds this ADR)

Four streams were researched (dialogue-state-tracking & classical dialogue management; production conversation frameworks; calibrated confidence / abstention; and our own codebase seams). The findings converge hard:

  1. Every production-trusted framework localizes the LLM to understanding and keeps a deterministic stack/policy for ordering and resume. Microsoft Bot Framework (dialog stack + two-tier InterruptAsync/AllowInterruptions consultation), Rasa CALM (an LLM "command generator" emits a closed command set — StartFlow/SetSlot/Cancel/Correct/Clarify — that a deterministic flow stack executes; CALM's docs explicitly contrast this with "ReAct-style agents where an LLM decides what should happen next" and name that the thing to avoid for "reliable, debuggable interactions"), and LangGraph (durable checkpointed interrupt() + Command(resume, goto)) all sit at the deterministic-disposition end. The frameworks that let a raw agent-loop own routing (OpenAI Agents SDK handoffs, Semantic Kernel) give up task interrupt/resume entirely — handoffs are one-way, no return — and surface only uncalibrated model-picks-a-tool routing. That end of the spectrum is exactly where teams retreat from when reliability matters, and our maestro is on a privileged path that dispatches real work orders.

  2. The classical literature already named and partially solved our four problems. Information-State dialogue management (TrindiKit) models "where are we" as an explicit, inspectable state with guarded update rules (preconditions → effects). RavenClaw's dialog stack + expectation agenda binds an incoming message only against currently-open contexts — structurally stronger than our flat "bind by signal" that got burned by the leak/leaking stemmer. POMDP belief-state DM carries a distribution over states and lets an explicit cost/reward decide ASK-vs-ACT — the principled form of "ambiguous → ask." Issue-Based (QUD) management models a pivot as answering a different open question, accepted by matching to the question stack rather than forced onto the current context.

  3. Calibrated uncertainty has a buildable recipe, and it is the genuine net-new work. No framework hands you a tuned RESOLVES/CONTINUES/PIVOTS/AMBIGUOUS classifier or a keyword-free escalation detector. But the calibration literature gives a defense-in-depth recipe — an explicit ambiguous class (CLAM), self-consistency / semantic-entropy voting (Wang; Farquhar et al., Nature 2024), P(True) self-evaluation (Kadavath et al.), and a risk-coverage threshold chosen asymmetrically so mutating decisions abstain conservatively (selective prediction, Geifman & El-Yaniv) — all measurable with ECE + risk-coverage/AURC + Brier/L2 on an adversarial labeled set.

Full streams with citations are in docs/planning/maestro-research/ (the four reports), and the source list is reproduced at the bottom of this ADR.

Decision

D1 — The fork is resolved: hybrid (c). The LLM proposes a closed command vocabulary; the deterministic ladder disposes.

The maestro is neither a hand-authored state machine that tries to classify intent with brittle rules (a), nor an LLM agent loop that drives routing itself (b). It is a hybrid (c):

A single LLM "dialogue-understanding" step reads the conversation + the open contexts and emits ONE schema-validated MaestroCommand from a closed vocabulary. A deterministic ladder — the existing 3-rung precedence ladder — executes that command, owns ordering, owns safety, owns resume, and is the sole site of any state mutation. The LLM never executes business logic, never picks the next handler, never mutates state; it only names what it thinks the message is, with calibrated confidence.

This is Rasa CALM's "command generator → deterministic flow stack," adapted to our ladder. It is chosen over the alternatives because:

The load-bearing rule: the command is a proposal; the ladder is the authority. A command can only ever cause a mutation by passing through a deterministic rung that re-checks safety and precedence. The LLM cannot route around the safety front.

D2 — The command vocabulary (what the LLM is allowed to say)

Per inbound message the dialogue-understanding step emits exactly one MaestroCommand, validated by JSON schema at the tool-call boundary (the structured-output-as-contract pattern — same as access-extractor.ts):

type MaestroCommand =
  | { kind: 'safety_raise'; severity: 'emergency' | 'optout' | 'abuse'; confidence: number; evidence: string }
  | { kind: 'resume';   targetContextId: string; relation: 'resolves' | 'continues'; confidence: number; slotDeltas?: Record<string, string>; evidence: string }
  | { kind: 'pivot';    fromContextId: string | null; confidence: number; evidence: string }   // → fall to full-route
  | { kind: 'clarify';  candidateContextIds: string[]; reason: 'ambiguous_relation' | 'multi_context'; question: string; evidence: string }
  | { kind: 'route_new'; confidence: number; evidence: string };                                  // no open context applies

This is a closed set (CALM's StartFlow/SetSlot/Cancel/Correct/Clarify mapped onto our domain). relation is the I3 vocabulary; clarify is the I3/I4 abstention output; evidence is the quote the command is grounded in (un-fakeable, gradeable). The LLM never emits "call handler X" — only what the message is.

D3 — MaestroDecision: the command + the disposition, as one inspectable record

The ladder executes the command and emits a MaestroDecision — the contract doc's record, extended from today's DispatchDecision. It is the single source of truth the action layer consumes, the top-of-screen graph renders, and the maestro eval lens grades. The new fields over DispatchDecision (per the codebase map's gap analysis):

interface MaestroDecision extends DispatchDecision {
  entered: 'safety' | 'resume-shortcut' | 'full-route'; // which rung disposed of the command
  command: MaestroCommand;                               // what the LLM proposed (incl. confidence + evidence)
  contextRelation: 'resolves' | 'continues' | 'pivots' | 'ambiguous' | 'none';
  openContextCount: number;                              // >1 ⇒ went through disambiguation
  boundContextId: string | null;                         // the context this message was bound to
  interrupted: boolean;                                  // safety preempted + PRESERVED an open context
  resumeRevalidated: boolean;                            // resume re-checked validity (I6), not blind
  abstained: boolean;                                    // command was clarify (asked rather than mutated)
}

recordMaestroDecision mirrors recordDispatchDecision (fail-soft, written after resolution so it reflects the actual outcome). The maestro eval lens reads it from the real source (the dispatch-log row, the WO binding, the active-context row) — never a mock — and grades the declared-vs-actual decision per stage.

The OPEN invariants — a buildable, evaluable mechanism for each

Each is tagged honestly: MECHANISM (a concrete buildable design + a fail-on-today eval), or OPEN (no buildable mechanism yet — closest approach named).

I3 · No-Guess-on-Mutation (ambiguity with calibrated confidence) — MECHANISM

Requirement. On an ambiguous relation, ask one clarifying question and bind nothing that mutates state. The hard part is detecting ambiguity reliably — classifiers say "continues" when they should say "ambiguous."

Prior art. POMDP belief-state DM (carry a distribution; flat mass = ambiguous; ASK-vs-ACT is a thresholded value decision). The calibration recipe: explicit ambiguous class (CLAM), self-consistency / semantic entropy (Wang; Farquhar Nature 2024), P(True) (Kadavath), risk-coverage thresholding (Geifman & El-Yaniv), temperature scaling (Guo).

Mechanism (buildable now). The dialogue-understanding step is sampled k=5 at T≈0.7 and emits the MaestroCommand each time. Compute a confidence from three independent signals: (a) the vote margin / label entropy across the 5 samples (a split between two mutating relations — e.g. resume:resolves ↔ pivot — is treated as ambiguous regardless of any single sample's stated confidence); (b) the model's verbalized confidence field; (c) a one-call P(True) cross-check on the top command. Fit a single temperature/logistic re-scaler on the combined score against a labeled set, and pick the abstain threshold off the risk-coverage curve, asymmetrically: a tight guaranteed-error bound on mutating commands (resume/pivot — e.g. <1% wrong mutations), looser on route_new. Below threshold → emit clarify (ask one question; mutate nothing).

The aggregation MECHANISM is built (2026-06-17): src/lib/domain/maestro/relation-verdict.ts — aggregateRelationSamples(samples, policy) → RelationVerdict is the PURE self-consistency-vote + asymmetric-abstain core: split-mutating → abstain, explicit-ambiguous vote → abstain, asymmetric agreement+confidence thresholds (mutating relations conservative), entropy as the semantic-entropy proxy. 10 unit tests incl. the overconfidence trap (3×resolves@0.95 vs 2×pivots@0.95 → abstain). The LLM sampling + the threshold tuning inject later — tuning is calibrated against the labeled adversarial set (ECE + risk-coverage), NOT guessed. So the mechanism is proven; I3 is "done" only once the eval tunes the threshold and a fail-on-today scenario passes.

Eval scenario (fails on today). Adversarial labeled set heavy on near-miss pivots and embedded-number access replies (the leak-204 family — "actually the AC's broken too", "fine to come in, 2 onward"). Today the system has no ambiguous output at all, so the scenario that asserts command.kind === 'clarify' (or contextRelation === 'ambiguous') on the genuinely-ambiguous cases fails by construction until the mechanism lands.

Metric. ECE + reliability diagram (calibration), risk-coverage / AURC with the operating point marked (guaranteed mutation-error at chosen coverage), over-ask vs missed-ambiguity rate. Pass bar: mutation-class risk under tolerance at acceptable coverage — it abstains-and-asks before it ever wrongly mutates state.

I4 · Single-Pointer-or-Disambiguate (multi-context binding) — MECHANISM

Requirement. One open context → bind; multiple → disambiguate. "Bind by signal" is the reply-attribution problem we already got burned on (the leak/leaking stemmer).

Prior art. RavenClaw's expectation agenda (bind only against the currently-open contexts' expected concepts, not a flat global matcher). SGD/TRADE (describe each open context in natural language; a description-conditioned binder generalizes to unseen phrasings with no keyword list; copy mechanism for free-text values). Rasa CALM pattern_clarification + action_clarify_flows. Contextual slot-carryover classes (none / in_current / in_history / in_cross_context).

Mechanism (buildable now). The dialogue-understanding prompt is given the open contexts only (each as a short natural-language description: "WO-204 — leak in unit 3B, awaiting your availability"), and the command's targetContextId must be one of them or null. When >1 context is open and no single command sample wins a clear majority, the ladder emits clarify with candidateContextIds — ask the human, never guess "most recent." Signal-match is a hint that biases the LLM's evidence, never the binder itself — this structurally retires the flat stemmer.

Eval scenario (fails on today). Two open WOs for one tenant + a reply that matches both on a stem ("the leak is back"). Today's flat attribution binds to one by signal/recency; the scenario asserts command.kind === 'clarify' with both context ids as candidates → fails today.

Metric. Per-context binding precision/recall on a "(message, open contexts) → correct binding" golden set; false-bind rate (the metric the stemmer bug would have caught).

I5 · Safety-Can-Emerge (keyword-free escalation) — MECHANISM (detection is the hard, net-new part)

Requirement. Any handler (incl. a resumed one) can raise to the safety front when severity appears with no emergency keyword ("water's through the ceiling now"). Two sub-problems: coverage (every handler must check, every turn) and detection (the keyword-free NLU).

Prior art. Bot Framework's two-tier interrupt / consultation (the active input consults the parent before consuming the turn). SGD's description-conditioned intent ("emergency-escalation" described in natural language → recall on never-seen phrasings, no keyword list). The detection itself is acknowledged across all four streams as genuine net-new work no framework provides.

Mechanism (buildable now — two layers). Coverage is structural and shippable immediately: the safety front is the first sample the dialogue-understanding step is graded on (safety_raise outranks every other command), and a single shared severityProbe(message, openContext) is the only severity check — lint/drift-guarded for coverage so no handler can forget it (the same drift-guard discipline as the spine-stamp construction invariant). Detection is the LLM command itself: safety_raise is one of the closed commands, with confidence + evidence, sampled and calibrated exactly like I3 (a false-negative is danger, so its abstain threshold is tuned for recall, accepting some over-paging).

Eval scenario (fails on today). A held-out set of keyword-free escalations ("the ceiling's sagging", "it's coming through the light fixture now") interleaved into an in-progress access-gather. Assert command.kind === 'safety_raise' and interrupted === true (the open context preserved). Today there is no keyword-free detector → fails.

Metric. Recall on the no-keyword escalation set (the load-bearing number — misses are dangerous), false-raise rate (the cost ceiling), and a coverage assertion (every handler routes through severityProbe).

I6 · Re-Validate-on-Resume — MECHANISM for the hook; OPEN for the learned "still-meaningful?" judgment

Requirement. Resuming a context re-checks it is still meaningful given what happened (esp. after a safety interrupt) — never blind-resume ("don't ask 'when are you free?' right after evacuating the unit").

Prior art. RavenClaw resumes on a pop by re-checking the agent's preconditions / completion criteria — the structural hook. LangGraph's interrupt()/Command(resume, goto) is the durable substrate, with a sharp hazard: on resume the node re-executes from its start, so any side effect before the interrupt must be idempotent — directly relevant to a maestro that can dispatch. The research is blunt that no mainstream system makes resume conditional on a learned "is this still relevant?" check — this is genuinely under-served.

Mechanism (the hook — buildable now). Resume is never blind: the ladder calls revalidate(context, sinceEvents) → { valid | stale | superseded } — a small structured-output classify over "what happened since this context was opened" — before re-entering a popped frame. superseded (a safety interrupt evacuated the unit) routes to a fresh handler, not the stale ask; stale (TTL backstop) closes the context. The TTL is a backstop on top of the relation classify (I3), never the primary signal.

OPEN part. The quality of revalidate — reliably judging "still meaningful?" after an arbitrary interrupt — is the same hard NLU as I3/I5 and has no proven mechanism in the literature beyond "another structured classify." We ship the seam + the safety-interrupt-superseded path (the one concrete, high-value case), and mark the general judgment OPEN with the closest approach (a calibrated revalidate classify, graded like I3).

Eval scenario (fails on today). Gas-leak interrupt mid-access-gather → after the emergency, assert the maestro does not resume "when are you free?" (asserts resumeRevalidated === true and the superseded path). Today there is no revalidation → fails.

I1 / I7 · Safety-First ordering & Precise detection — ordering SOLVED, fuzzy detection MECHANISM

Ordering is solved structurally (I1 runs first; I7's rating case landed — see below). The fuzzy detection behind both (an emergency with no keyword, a precise-but-not-anchored intent) is the same calibrated-command mechanism as I3/I5: the safety_raise command sampled + calibrated for recall. There is no separate mechanism — I1/I7 fuzzy detection is the command layer.

What is already proven and landed (the SOLVED half)

Implementation plan (rung by rung, each lens-graded, each with a fail-on-today eval)

The contract's plan, refined with the mechanisms above. One source of truth — each step REPLACES the prior behavior in the same PR; no parallel paths.

  1. MaestroDecision schema + the maestro eval lens. Extend DispatchDecision → MaestroDecision; recordMaestroDecision (mirror the fail-soft writer). Add the maestro lens to the golden checkpoint engine, reading the real dispatch-log row. Eval: the leak-204 access checkpoint asserts entered: 'resume-shortcut' / contextRelation: 'resolves' — red today, green after I8 routing + the lens. (This is the immediate unblock the handoff described — the I8 fix already landed; the lens makes it gradeable end-to-end.)

    DONE 2026-06-17 — wired + LIVE-BENCH CONFIRMED (incl. a placement correction + a timing-bounded finding). Bench-owner (2c43af68) ran leak-204 on real prod rows: noneOfReasons GREEN — the rating-hijack is dead (no state_first_rating for the tenant). The anyOfReasons half first came back red, correctly diagnosed as a checkpoint-placement bug, not a behavior regression: it was wired on the dispatch stage (turn 3), but state_first_resident_answer doesn't fire until the relay roundtrip (turn 6). Fix: moved the maestro checkpoint onto relay.diagnostic-roundtrip (turn 6) — both halves green there.

    ⚠️ Timing-bounded finding (bake into any future maestro fence): state_first_rating is emitted by BOTH the hijack AND the legitimate end-of-flow 5-star rating (turn ~11). So a person-scoped noneOfReasons fence is only valid in the mid-flow window (after the contested reply fires, before the legit emission). NEVER grade it at end-of-flow (false-reds on a real rating) or too early (its anyOfReasons half hasn't fired). The one correct home is the turn where anyOfReasons first holds. Documented on the MaestroCheckpoint type + the leak-204 cell. Built end-to-end:

    • The ddr lens (checkpoints.ts:evalDdr) gained noneOfReasons — the negative-routing primitive (unit-tested red-on-old / green-on-fix).
    • The reader-scope discovery (two iterations): the per-turn maestro assertion CANNOT use the per-WO reader (the hijack stamps its state_first_rating row against the stale completed rating-target WO, so a per-WO ddr checkpoint is blind). A first cut went person-scoped — but the reviewer caught a sharper bug: the person partition accumulates every turn/scenario/run for 90 days, so a prior run's legitimate turn-11 rating leaves a persistent state_first_rating row that would false-red the next run's noneOfReasons (the timing-window placement doesn't save it across runs). The correct scope is per-TURN via the dispatch traceId: the hijack IS that turn's routing decision (same traceId, even when bound to a stale WO), so a trace-scoped read catches it AND is immune to cross-run accumulation. The pipeline-test ptest_… requestId IS the dispatch-log row key, so result.traceId keys the read precisely.
    • A maestro checkpoint kind (MaestroCheckpoint, system 'maestro') + evalMaestro lens (TURN-PRECISE sibling of evalDdr), reading the existing getDispatchDecisionsByTrace (wired into CheckpointReaders.listDispatchDecisionsByTrace; the turn's traceId threaded through EvaluateTurnCheckpointsArgs from both eval-activity call sites). Fail-closed when no traceId (never a silent green — the verify-the-real-source rule). Assertions: anyOfReasons / noneOfReasons / minCount + enteredEquals — grades WHICH rung won the turn (safety_preempt_* → 'safety', state_first_resident_answer → 'resume-shortcut', else 'full-route'), the contract's MaestroDecision.entered vocabulary, via the pure exported deriveEnteredRung.
    • Threaded through the golden compiler (PropflowExpectSpec) and wired onto leak-204's dispatch stage (post-create, so the WO resolves the personId): { system: 'maestro', noneOfReasons: ['state_first_rating'], anyOfReasons: ['state_first_resident_answer'] }.
    • Unit-tested (4 maestro cases — person-scope catches the cross-WO hijack, green-on-fix, fail-closed-no-person, read-throws); 68/68 checkpoint tests, tsc clean.

    Remaining: one live leak-204 bench run to confirm the maestro checkpoint goes red-on-old / green-on-fix against REAL dispatch-log rows — owned by the bench-owner session (warm env). The full MaestroDecision schema fields (entered/contextRelation/abstained) arrive with the command layer (steps 3–5); this lens grades the routing-reason layer now.

  2. I1 safety front — explicit, un-skippable, intra-ordered + the shared severityProbe (coverage drift-guard). Eval: gas-leak-mid-access-gather → safety_raise + interrupted: true.

  3. The dialogue-understanding command step (D2) + I2 fresh-shortcut TTL gate. The single LLM step that emits MaestroCommand; the resume rung gates on TTL as a backstop. Eval: a 3-days-later stray → contextRelation: 'none' (stale context closed, not resumed).

  4. I3 + I4 — the calibration layer (k-sample + vote-entropy + P(True) + risk-coverage threshold) and the disambiguation rung. Eval: the adversarial ambiguity/multi-context sets above (ECE + risk-coverage + binding precision).

  5. I5 + I6 — the upward safety_raise escalation exit and revalidate on resume. Eval: keyword-free escalation recall; gas-leak-then-no-blind-resume.

Each step ships its invariant's eval scenario written to fail on the prior behavior — so the ladder is proven, not asserted.

How the eval grades it (the can't-lie standard)

The maestro lens reads the real source — the dispatch-log routedTo/reason, the WO binding, the active-context row — never a mock, never PropFlow's own optimistic echo (the "verify the real source, never self-validate" rule). Calibration mechanisms add three offline metrics on a labeled adversarial set, recomputed weekly and re-fit on drift: ECE + reliability diagram (calibration), risk-coverage / AURC (guaranteed mutation-error at the operating coverage), and Brier/L2 over the context distribution (the overconfidence lever the DST/DSTC literature identified). An invariant is not done until its scenario — written to FAIL on the prior behavior — passes.

What this does and does NOT solve (blunt)

Does solve / settle:

Does NOT solve (still genuinely hard):

So the claim is precisely: "settles the architecture and makes the remaining judgment buildable + measurable" — not "solves conversation routing."

Alternatives considered

Open questions (carried from the contract, now scoped)

Sources

Production frameworks: Bot Framework — handle user interruptions, Adaptive dialogs inputs / AllowInterruptions; Rasa CALM, Command Generator, Conversation Repair patterns, TED policy; LangGraph interrupts; OpenAI Agents SDK handoffs; LiveKit turn detection, Pipecat. Dialogue state tracking / classical DM: Information State approach (Traum & Larsson), RavenClaw (Bohus & Rudnicky), POMDP-based SDS review (Young et al. 2013), Schema-Guided Dialogue (Rastogi et al.), TRADE (Wu et al.), DST survey, Contextual slot carryover, GoDiS/IBiS (Larsson). Calibration / abstention: Guo et al. — calibration & temperature scaling, Tian et al. — Just Ask for Calibration, Kadavath et al. — LMs (mostly) know what they know / P(True), Geifman & El-Yaniv — selective classification, Farquhar et al. — semantic entropy (Nature 2024), Wang et al. — self-consistency, Kuhn et al. — CLAM, Aliannejadi et al. — clarifying questions.