PropFlow Technologies · Engineering
The system of work for residential property management: one AI operator — Clara — who holds goals, acts within policy, proves every outcome, and learns each lesson once for every property. This document is the engineering counterpart to the product vision: what we run today, the principles we build by, the target system, and the path from hundreds of units in one state to thousands of units in many.
Our product vision is one sentence: the PMS remains the system of record; Clara becomes the system of work. A property-management company delegates outcomes to Clara — lease the unit, resolve the issue and close the loop, renew the tenant, collect what's owed — and Clara works them to completion across SMS, email, and voice, escalating only the decisions that genuinely need a human, and converting each human answer into standing policy so the same question is never asked twice.
That product only exists if the engineering underneath it earns a specific kind of trust: an agent that acts in the real world, on behalf of regulated businesses, toward people's homes. This document defines the architecture that earns it. It is written to be read by our own team first, and by any engineer, client, or diligence reviewer we choose to share it with. It states principles we will not compromise, describes the system as it actually is — including its faults, with evidence — and lays out the target system and the sequence to get there.
Three inputs shaped this revision: a systematic mining of our own three months of production history (4,393 merged PRs, five written incident root-causes, ~170 catalogued failure findings); a survey of how the strongest agent companies in adjacent high-stakes verticals build (Decagon, Sierra, Intercom, Harvey, Salient, Gradient Labs, and our direct peer EliseAI); and a working proof-of-concept that rebuilt our leasing loop on the leading agent framework and replayed 262 real conversations through it. Where a claim in this document has a number, the number traces to one of those sources or is labeled an estimate.
Eight principles govern every design decision in this document. Each exists because we, or a company we studied, paid for its absence. Each is stated with what it forbids — a principle that forbids nothing decides nothing.
No component may report an outcome it cannot prove. A tool that sends returns a carrier receipt or it returns pending. A test that didn't run reports "skipped," never green. A dashboard that can't read the bill shows no number rather than $0. Our three-month history shows one disease behind our worst incidents — success claimed without proof — appearing independently in tools, CI, auto-merge, and reporting. Clara's honesty layer is one instance of this principle; the architecture generalizes it.
Forbids: returning success from side-effecting code without downstream confirmation; rendering a skipped check as passing; any "fire and forget" send to a person.
Cheap, explainable checks run first and handle everything they can: routing lanes, life-safety pre-empts, schema validation, pattern tripwires, rate limits. A small number of calibrated LLM evaluators handle only what genuinely needs judgment. This is also the converged industry pattern (Decagon, Sierra, and the 2026 reliability literature agree), and it is why our routing ladder — most turns resolved without any model call — is a strength to extend, not legacy to replace.
Forbids: reaching for a model where a rule suffices; guard logic whose behavior can't be explained to a regulator after the fact.
Everything Clara is allowed or forbidden to do lives in a versioned policy store, in three tiers: state law (externally sourced facts — notice periods, deposit rules, disclosure duties, consent regimes — never inferred by a model at runtime), property policy (each operator's choices inside the legal envelope — where "the answer becomes policy" lives), and code (which is never state-specific; it consults the tiers, refuses or escalates when a rule is missing, and logs which policy version backed each decision). Every policy is addressable by ID and emits telemetry when it fires, so "how often did policy X trigger last week, and what did it catch?" is a dashboard query, not an investigation. Because a central policy layer is also a tampering vector, it is versioned, write-restricted, and drift-monitored.
Forbids: hardcoded legal rules; prompt-only compliance; policy changes that require a deploy; a policy firing without a trace.
Safety lives next to the side effect, not in the reasoning. The model may think freely; the tool that sends an email validates the recipient, the content class, and the rate; the tool that writes to the PMS validates the record scope; irreversible or high-stakes actions carry needsApproval and pause for a human. An agent-level guard cannot predict every tool call — the tool boundary can check every one that matters, deterministically.
Forbids: trusting model-emitted identity or scope inside a tool handler; any high-stakes side effect without an approval class; safety logic that only exists in the prompt.
There is exactly one interactive agent loop; every channel enters it and every reply leaves through it. Background lanes decide who to contact and when — they never run a second conversation engine. Every person, conversation, and interaction has one canonical record that all channels read and write; identity is resolved in one place. Six of our recurring failure classes (duplicate records, cross-channel amnesia, language resets, thread splits) are downstream of violating this principle piecemeal.
Forbids: per-channel context stores; a second code path that composes tenant-facing text; any subsystem creating person records outside the canonical resolver.
Clara acts within policy; outside it she doesn't guess — she asks, and the answer becomes property-tier policy so she never asks that question again. The teach loop is therefore a first-class system with a gate: answers touching protected categories (income source, familial status, disability, and their state-specific kin) pass a jurisdiction-aware review before they lock in — because the fastest way to a consent decree in our industry is one wrong scripted answer replicated across every property (Harbor Group/PERQ, 100+ properties, 20 states).
Forbids: silent generalization of a one-off concession into standing policy; any protected-category answer entering the policy store without review; re-asking a settled question.
Production traces are the eval suite's food supply. Every incident, every human 👎, every guard catch becomes a permanent regression case within 24 hours; the suite grows monotonically and gates merges. Green must mean tested: a check asserts its own validity — right identity, right environment, actually executed — before its verdict counts. Our own history is the argument: five separate testing layers each missed the same bug because none verified it was testing the real thing.
Forbids: skip-rendered-as-pass; tests that assert on their own mocks; an incident closed without a regression case; eval results that live in a log nobody can trend.
We own our prompts, our context assembly, our control flow, and our policy layer — they are the product. We buy or adopt what observes the system: score storage, dashboards, trace tooling — over open telemetry standards so the vendor stays swappable. This is the documented practice of every serious vertical agent team (thin proprietary guard/eval layers over their own loop; heavy frameworks adopted selectively or not at all), and our own framework proof-of-concept reconfirmed it empirically.
Forbids: an orchestration framework in the conversation path; a vendor between Clara and her model calls; instrumentation that only one vendor can read.
Honest accounting first. The platform below is real, in production, and handling real residents across three properties today. Its bones — direct model API, durable workflows on Temporal, a single-table store, one interactive loop behind deterministic routing — match where the industry landed after two years of framework churn. Its faults are equally real, they are measured, and the rest of this document is organized around closing them.
Seven structural fragilities (F1–F7) and eight recurring failure patterns were catalogued from code inspection and the three-month mining pass. In brief — the full evidence sits in the appendix:
A dual module tree that resolves the same import to different code in different builds, and a ~6,000-line single-function loop whose guard order is load-bearing but expressed only as code position. Neither is visible until it bites; both have bitten.
~15 unrelated LLM callers share the tenant loop's circuit breaker: five failures in a low-stakes lane fail-fast every tenant conversation for 30+ seconds. Blast-radius isolation is a P4-class fix.
Replies are persisted before the carrier send, with no reconciliation if the send fails — history can show words a tenant never received. And a total loop failure produces deliberate silence with no floor except the life-safety lexicon. Both are P1 violations at the most important boundary we have.
The LLM-graded guard chain does not exist on calls; one deterministic substitution is the only inline protection. Voice is our fastest-growing channel and our least-checked.
The failure patterns tell one story. The largest cluster is the say-do gap — tools and reports claiming success for actions that never happened (an application link "sent" while broken for 19 days; an escalation tool that crashed on its first line for 15 straight days while telling tenants help was coming; 42% of "connected" call transfers never reaching a human). The second is green-isn't-tested — a nightly behavior eval that rendered green on 128 of 130 runs while silently skipped in all but 2; unit tests that mocked the exact call that was broken. The third is watchers that die with their session — completion promises held in ephemeral processes rather than durable jobs. Patterns four through eight (grounding failures, fragmented identity, cross-channel context loss, rules restated instead of enforced, voice misrouting) each trace to a specific violated principle above. This is why the doctrine reads the way it does.
Deep-dive on the renewal system specifically (the Sept 3 duplicate-outreach incident, the ranked flaws, and the keep/fix/refactor options) lives on its own page: Renewals architecture: keep, fix in place, or refactor.
The target system is the current system re-organized around the doctrine, plus one genuinely new layer. We describe it as five planes; each subsection states the design, what exists today, and what must be built.
The hand-rolled loop stays. This decision is now triple-sourced: two independent framework studies, the 2026 reliability literature ("most serious agent teams build thin proprietary layers over their own models"), and our own proof-of-concept, which rebuilt the leasing loop on LangGraph, replayed 262 real conversations through it, and found the one thing the framework provides for free — a built-in human-interruption primitive — survives only within a single process, while our Temporal-based escalation survives crashes, deploys, and multi-day waits. What changes is the loop's shape:
This is the largest new investment, and the direct generalization of P1 and P4. Every tool that touches the outside world adopts a common contract:
The receipt contract. A side-effecting tool call writes a pending intent record before acting; the action produces a downstream confirmation (a carrier message ID, a PMS write-back read, a page acknowledgment); only confirmation converts pending to done — and only "done" may be reported as success to the model, the transcript, or a human. Anything pending past its class timeout raises an alert naming the tool, the person affected, and the promise made. Delivery status joins the conversation record itself, so a transcript reader — human, judge, or harness — sees what a tenant actually received, not what we composed (closing F4). Total-failure silence (F5) gets a deliberate floor: a policy decision, made once, about which failure classes owe a tenant a deterministic acknowledgment.
At the same boundary, per P4: recipient and scope validation move into the send/write tools themselves; model-emitted identity is never trusted where verified identity exists (already our practice in the tool dispatcher — extended to every handler); and a defined class of actions (lease-term statements, payment-adjacent operations, anything irreversible) carries needsApproval and pauses through the same Temporal-backed escalation the rest of the system uses. The three worst say-do offenders from the incident record — PM forwarding, application links, renewal escalation — convert to the receipt contract first, because each has already silently failed against a real person.
Today, "what Clara may do" is scattered: prompt rules, guard code, hardcoded constants, and a teach loop that writes straight to production knowledge. The target consolidates enforcement into the declarative guard pipeline (outbound) and the tool boundary (actions), both reading one policy store:
| Tier | Contents | Change process |
|---|---|---|
| State tier | Facts of law per jurisdiction: notice periods, deposit rules and deadlines, late-fee caps, screening and source-of-income rules, required disclosures, recording-consent regime, AI-disclosure duties, collections restrictions | Externally sourced and versioned; a named owner runs legislative change-detection — a rules layer without an owner rots at the speed of legislatures. Never model-inferred at runtime. |
| Property tier | Each operator's choices inside the legal envelope: pet terms, concessions, screening criteria at or under state caps, escalation routing, quiet hours. This is where a taught answer becomes standing policy. | The teach loop, gated: protected-category answers require jurisdiction-aware review before locking in (P6). Every entry carries provenance — who taught it, when, from which conversation. |
| Code | The enforcement machinery itself — never state-specific. | Normal engineering process. Logs the policy ID and version behind every allow/deny/repair, producing the audit record that turns "we told the model not to discriminate" into an actual defense. |
Every policy is addressable by ID, fires with telemetry, and appears on a dashboard ("policy X fired 14 times last week; 12 repairs, 2 holds"). We build this as our own thin module — the option an external policy engine (OPA/Cedar) exists for is externally-managed policy, which we do not need yet; the decision reverses cheaply because the store is data. And because a central policy layer concentrates risk, it inherits infrastructure-grade controls: versioned changes, restricted writes, drift monitoring.
Clara's answers are only as good as what she is grounded in, and grounding fails two distinct ways that must be measured separately: invented (the model asserts something no source supports) and stale (the model faithfully cites a source that is out of date). The second is the sneakier one — Intercom's Fin team reports sub-1% hallucination and still had to audit and update 700+ help articles before trusting it internally, because a grounded answer from a stale article is confidently wrong with a perfect citation. Our equivalents of that article base are PMS-derived facts: availability, pricing, lease terms, office hours. The knowledge plane therefore carries:
website | rent_roll | manual); the target adds a freshness horizon per fact class and a staleness monitor that flags — and where high-stakes, quarantines — facts past their horizon. "Confidently wrong from outdated data" becomes its own tracked failure class, distinct from fabrication.This is the layer that decides whether we scale. Today a founder personally reads every conversation — the right choice at three properties, an impossibility at three hundred. The target is a pipeline in which every conversation is scored, the human queue is bounded by policy rather than volume, every human verdict compounds into the regression suite, and a score regression can stop a merge. Roughly 60% of it already exists in parts (machine grades, the Grading Playground's ranked desk, real-traffic replay harnesses, a nightly two-model adversarial review); this section wires the parts into one system and adds what is genuinely missing.
| Gate | What runs | Disposition |
|---|---|---|
| Every agent-surface PR | Deterministic floor: schema and tool-shape validation, policy hard rules (fair housing and collections at zero tolerance), the no-literal-names prompt lint | Blocking, seconds |
| Every agent-surface PR | Paired regression: 100–300 stratified real-replay cases scored on the branch vs main by our subscription-run judge; fails on a defined paired drop; posts per-case diffs to the PR — the eval receipt | Blocking, minutes |
| Every agent-surface PR | Adversarial red-team: prompt-injection and data-exfiltration probes, including indirect injection — hostile instructions embedded in inbound emails, maintenance tickets, and CRM notes, the exact surface Clara ingests all day. (Our replay already caught a staff-impersonation attempt in the wild; the POC refused it — this gate makes that refusal a regression-tested property.) | Blocking on threshold |
| Nightly | The full corpus (1,000+ cases and growing via production promotion), batch-priced; scores land in the trend store; alerts on week-over-week drift, not just absolute floors | Advisory + dashboard |
| Continuous | Post-call voice scoring (async, zero in-call latency) and production sampling into the same trend store — one quality view across SMS, email, and voice | Advisory; feeds the corpus |
Two standing repairs make the gates honest (P7): a skipped check may never render as passing — today our flagship behavior sweep has produced no completed run in its last 40 attempts while showing green — and our most rigorous scenario platform (fail-closed, human-approved judging) graduates from post-merge advisory into the PR gate for the scenario classes it covers. Every production incident becomes a permanent eval case within 24 hours, mechanically tracked.
Scores, trends, and datasets live in a bought lens (P8): our judge computes every score on our own subscription runners; the platform stores and trends them over open telemetry, so the vendor stays swappable. The specific vendor is Decision D1 below.
Decided 2026-08-18: D1 went the other way for this one layer — no vendor. Quality scores live in our own append-only store with our own dashboard UI; Langfuse (and Braintrust/Arize) served as design inspiration only, and Langfuse itself stays internal LLM-call tracing plumbing, never the score sink or the operator surface. P8's "buy the lens" still governs general tracing; it does not govern quality scoring. Detail in §10, decision D1.
Clara's honesty layer — the guard chain referenced throughout §3.1 and this section — has its own four-month history worth telling in full, because the doctrine and target architecture above generalize lessons this layer paid for directly. The complete inspection (17 code agents plus 14 research agents; every code claim verified against the repo) lives at the full honesty-layer inspection page; this section distills it into the source of truth for what it found and what changes.
The May 27 reckoning. A sweep of 5,036 real traces found 34 blocks — about 79% false positives — including a tenant who had already signed still tripping the guard on "take care!" for 60 days. The fix narrowed scope to open renewals only. The lesson (every widening of scope had produced a false-positive incident) is why guard scope is something the roadmap treats as a thing to prove, not assume.
The Aug 13–16 saga ("P8"). A new guard ("never fabricate confirmations or slots") shipped on 36 rows of synthetic evidence, with a broken exemption check reading a field the tool never actually produced. Over the next two days it told three real Camellia customers "nothing is scheduled yet" when something was — a prospect with a real tour an hour out, a correct pricing answer, and someone trying to cancel a real 2pm appointment. Fede ordered the rollback on Aug 15; the guard family was fully reverted, and a mandatory replay gate now applies to any future guard change that can replace customer text: replay ≥30 days of real conversations, hand-label every real-person would-block, before it can merge.
The natural experiment. Same team, same week, same incident family, two designs, opposite outcomes — the cleanest evidence for the redesign below: the blocking guard caught 0 of 28 real fires; the repairing guard ("make it true" — fire the real escalation instead of just describing one) had 0 false positives on 166 real people.
Why it broke structurally, not just once:
needsApproval appears nowhere in the guard code.The proposed design — the direct ancestor of §3.1's declarative pipeline and §3.3's policy plane — puts one checkpoint on every outbound message: a deterministic policy registry, one shared "what is true right now" ledger read fresh off the PMS, and a small trajectory-aware judge with a narrow trigger and a gold set behind it. Every check returns a three-way verdict — pass, repair (fire the tool that makes the claim true, one attempt), or hold for human review — with hard block reserved only for pre-tuned, high-confidence deterministic hits. Voice is reconciled after the call against the same ledger, corrected via callback or text — never inline, because the founder ruled zero added latency on live calls is non-negotiable (2026-08-16, see the standing product rule). Four increments carry the migration, each gated on its own replay-number proof: (0) fix the outbound-composition contract that let a correct answer get silently dropped across tool-calling rounds; (1) re-land the shared ledger with zero verdict drift; (2) the three-way verdict plus a tool-execution gate; (3) the judge, narrow-trigger, against a ≥200-case gold set; (4) voice reconciliation and retiring the six-snapshot scatter.
Four decisions from this redesign are still open — see H1–H4 in §10 Open decisions below. Where this shows up in the tracker: Phase 1's source-of-income tripwire and teach-loop gate rows, and Phase 3's guard-chain-declarative-pipeline row (§ Tracker).
A same-day companion investigation to the honesty layer above: a 30-agent, every-claim-code-verified sweep of everywhere AI-written or staff-written text reaches a resident or prospect, and whether a fair-housing screen sits in front of it. Full record at the fair-housing screening architecture page; companion to the Overnight Quality Campaign report.
Already live, not parked: the staff-answer tripwire (§4.5 and the teach-loop review gate, PR #5891, merged Aug 19) screens every staff decision reply before it reaches a person or becomes policy; the service-animal rule (PR #5896) stops Clara demanding proof of a trained service animal beyond HUD's two permitted questions.
What the sweep found: 21 confirmed unscreened paths. The themes, by risk:
| Theme | What happens today | Risk |
|---|---|---|
| Maintenance texts | A tech's free-text update is AI-rewritten and texted to the resident — nothing screens it | HIGH |
| Follow-up campaigns | AI-personalized outreach and re-engagement messages (text and email) skip the screen entirely | HIGH |
| Residents are exempt | The one live screen only runs for prospects; any reply to a known resident bypasses it — and accommodation requests mostly come from residents | HIGH |
| Voice reads taught policy | Phone-call Clara answers from property knowledge, including anything staff taught before the tripwire existed — never screened | HIGH |
| Old taught policies | Everything taught before Aug 19 is still on file unscreened and feeds every conversation | MEDIUM |
| Tour-reminder inserts | Operator-written "arrival instructions" go into reminder texts verbatim | MEDIUM |
| Paired-name parity | One live paired test showed different offers to "Emily" vs. "DeShawn" — a single observation, needs a proper repeated trial before it counts as a finding | INVESTIGATE |
Why this is parked instead of patched. Patching 21 call sites one by one bolts a screen onto every lane forever — every new feature reopens the question. Four structural questions are unresolved: where the screen sits (per-lane today vs. once at the send chokepoint, where every message passes anyway); where policy lives (property-level only today — the Gera conversation raises a portfolio-default → property-override hierarchy); where law lives (fair-housing law varies by state — vouchers are protected in Colorado, not everywhere — and jurisdiction isn't modeled anywhere; today the whole portfolio is Colorado so nothing is wrong yet, but nothing would catch it changing); and screen the source or the mouth (what's taught vs. what's said are different jobs — today we have one check apiece, in one lane each).
Fede parked this Aug 19 ("this needs more thinking, can we park it") and handed it to a different workstream Aug 20; three options are on the table for whoever picks it up — A, chokepoint screen (one gate at every send path, audience- and jurisdiction-aware, recommended draft), B, patch the high-risk lanes only (fastest, leaves the structural question open), C, fold into the policy-hierarchy design with Gera (slowest to first protection, cleanest end state) — with A and C explicitly compatible (A now for protection, C as the model it later plugs into). One carve-out shipped independently Aug 20: the voice callback may read taught policies gated to rows that already passed the screen — a filter on existing screening, not a position on any option above. A chokepoint screen's estimated cost: about $0.001 and 0.1s per outbound resident/prospect message, fail-closed (a screening outage pauses outbound sends) — the same posture already accepted for the staff-answer lane.
This page carries no decision widget of its own — it is a parked handoff brief, not yet a founder vote. Cross-references: Phase 1's source-of-income tripwire and teach-loop gate rows in the tracker below; §4.5's honesty-layer redesign, which proposes the same single-chokepoint pattern as Option A here.
Scaling this business is not a bigger instance of the current system; it changes three assumptions at once — one property becomes many firms, one state's law becomes a matrix, and one PMS becomes a portfolio of them. The architecture answers each explicitly.
The policy plane's state tier (§3.3) is the mechanism; the operating discipline around it matters as much: a named owner for legislative change-detection per active state; every rule versioned with its source; conversations logging which rule version they consulted; and the deliberately uncomfortable rule that a missing or ambiguous state-tier entry produces refusal-and-escalation, never model improvisation. Voice adds two state-sensitive dimensions of its own — recording-consent regimes (our research found the two-party-consent state list genuinely contested across sources; it gets one-time legal confirmation before voice expands into contested states) and state AI-disclosure duties, which differ enough that a single global disclosure behavior is guaranteed non-compliant somewhere.
Our system of record is also now a competitor: AppFolio ships its own AI and gates third-party API depth — payments, screening, and insurance are walled off entirely. The response is a port boundary: a PMS adapter interface (read first, write second) behind which AppFolio is one implementation. This is defensive architecture — it caps platform risk and it is the technical substance behind a sales claim none of the PMS-native AIs can make: Clara works across whatever a portfolio actually runs.
A new property inside an existing firm is largely self-serve today; a new firm still requires an engineer for phone routing, lease-term configuration, and voice enablement — each currently a code or console change. Those gaps close into the onboarding flow, and every go-live follows a quality-gated ramp adapted from the best published playbook in a regulated vertical (Gradient Labs): begin at low volume with 100% human review, expand 25% → 50% → 100% as measured quality holds, with the compliance track running in parallel from day one — never as a pre-launch afterthought. The first two weeks of any new firm re-run our failure-taxonomy pass on their traffic, because a taxonomy built on one operator's policies will not cover another's without checking.
An agent that acts needs operations-grade reliability discipline, measured with agent-grade metrics. Pass/fail eval rates are necessary but not sufficient; the numbers that describe whether Clara is a trustworthy employee are:
Durable operations close the loop: watchers live in scheduled server-side jobs with queryable state (never in ephemeral agent sessions — the most-repeated operational failure in our three-month history), merges flow through a native queue, and the CI pipeline itself is reshaped (native-architecture runners, changed-path job selection) rather than audited into temporary savings a thirteenth time.
We claim Clara is safe and effective; this section is how we know, and it is deliberately the part we intend to publish. No competitor in our vertical publishes an evaluation methodology — searched directly; the strongest ones disclose nothing — which makes proof-in-public both a trust asset and a differentiator we can hold.
Sequenced against the business: Camellia deepening now, Yale onboarding, external firms in sale, multi-state to follow. Each phase names its architecture deliverables and the exit criteria that gate the next. Dates are targets, not promises; exit criteria are the commitment. The plan below is as written 2026-08-18; founder decisions since then are recorded inline, and §09’s live tracker carries per-item status.
| Phase | Architecture deliverables | Exit criteria |
|---|---|---|
| Phase 1 — Receipts & truth now → ~2 weeks | Receipt contract on the three worst say-do offenders (PM forwarding, application links, renewal escalation) with stuck-pending alerts · source-of-income deterministic tripwire + the known bad conversation pinned as a regression · teach-loop review gate decision for protected categories · trajectory-judge input fixes, replay re-run, policy gate armed repair-only on text at one property (on ≥80% measured precision) · ElevenLabs native test suites on · score store stood up (D1) with the first trend dashboard · judge calibration report v1 · skip-never-green fixed in CI Updated 2026-08-19/20: receipts on all three offenders shipped (per-item status in §09) · gate arming ON HOLD — the founder decided the pre-send gate stays off for now · D1 decided 2026-08-18 = in-house (the MachineGrade store + Quality Desk are live; the trend dashboard is not started) · calibration report v1 done · ElevenLabs native suites are currently not running (see the eval inspection). | Zero silent-success paths on the three converted tools; a quality trend line exists; Re-cut 2026-08-20: the gate criterion is superseded — the founder decided the pre-send gate stays off, so Phase 1 exits on the receipts (done) plus a quality trend line (open). |
| Phase 2 — Gates & flywheel ~30 days | PR gates blocking on agent surfaces (deterministic floor + paired regression + red-team incl. indirect injection) · distiller + corpus-runner finished (👎 → regression case, automatically) · native merge queue + durable server-side watchers · monthly clustering job v1 · the founder's 100-trace taxonomy pass · voice literal-name lint · the two unsynced voice agents onto CI sync Updated 2026-08-20: D3 decided = blocking gates (build authorized; which suites gate first is an open call on the eval inspection) · the distiller→corpus→promotion pipeline is BUILT and deliberately held (four green “do not merge” PRs) — founder 2026-08-20: it stays held until agentic quality and testing significantly improve. ⟶ Superseded as the active plan by the Quality System Plan (2026-08-20) — same goals, resequenced with the input layer added; the gap-by-gap status now lives on the Eval & Testing Roadmap; §09's tracker keeps per-item status. | A prompt regression demonstrably cannot merge; every human 👎 lands in the corpus without manual work; zero session-scoped watchers remain |
| Phase 3 — Policy plane & hardening ~60–90 days | Policy store v1: state tier for TX + first expansion targets, property tier migrated from scattered config, per-policy telemetry + dashboard · jurisdiction review gate live on answer-becomes-policy · pricing-data isolation invariant + tests · guard chain extracted to the declarative pipeline · breaker isolation · F5 floor decided and shipped · staged prompt rollout on per-property flags Updated: D4 decided 2026-08-18 — the state tier starts when the first out-of-state contract signs, not on the calendar. | Every allow/deny/repair logs a policy ID + version; a policy change needs no deploy; a protected-category answer cannot lock in unreviewed |
| Phase 4 — Multi-firm readiness ~Q4 2026 | Client-scoped quality dashboard · onboarding wizard closes its engineer-required gaps · quality-gated go-live ramp as standard · PMS adapter port (read side) with a second PMS spike · voice simulation + load testing · published evaluation methodology v1 + contractual data portability · first CS/implementation hire scoped Updated 2026-08-20: D5 decided 2026-08-18 — the methodology page will NOT be published (deliverable dropped). Timing is pulling forward: the Yardi partner contract and ConAm onboarding work began in Aug 2026, ahead of this phase’s ~Q4 slot for second-PMS/multi-firm items. | A new firm reaches first-conversation without engineering hand-edits; their ops lead can see their own quality dashboard; the methodology page is public |
| Phase 5 — Multi-state operation Q1 2027 | State tier populated + legally reviewed for each launch state (recording-consent and AI-disclosure confirmed) · per-state disclosure behavior wired · dual-tree module resolution retired · loop decomposition continues behind the regression suite's protection · Temporal-wrapped conversation turns where duty-of-delivery demands it | A new state is a data + review exercise, not an engineering project; the launch checklist is boring |
Live tracker · updated as work lands. Verified 2026-08-19 by a 7-agent repo sweep against the roadmap above (§08); every row's evidence traces to a file, a PR, or a recorded founder decision — never inferred. Sibling page for the product/coworker surface (email, voice, staff actions): the Clara-as-a-coworker tracker.
Zero silent-success paths on the three worst say-do offenders; a quality trend line exists; the honesty gate is live on one property within its target false-positive rate.
| Item | Status | Evidence |
|---|---|---|
Receipt contract — PM forwardingforward_to_property_manager only reports delivered when SendGrid actually accepted the send; a distinct "nobody notified" result replaces the old blind await-then-log-success bug. Caveat: this is a synchronous honesty check, not an async pending→confirmed outbox — that fuller state machine exists only for tour SMS (row below). | DONE | handle-forward-to-property-manager.tsCI guard: tool-result-honesty-invariant.test.ts |
Receipt contract — application linkssend_application_link claims success only after a real send outcome, with an atomic claim lock against double-send. This was the originating incident for the CI honesty guard above. Caveat: one code path writes the "confirmed" timestamp before checking the underlying send outcome — flagged as a possible bug, not independently re-verified today. | DONE | send-application-link.ts |
Receipt contract — renewal escalationrenewal_escalate throws instead of silently "succeeding" when the PM email bounces or is suppressed; a separate "tracked matter" lane confirms a human was actually reached before anything reports success. | DONE | agents/clara/lib/agent/tools/renewal/handlers.tstest: renewal-escalate-honest-failure.test.ts |
| Stuck-pending alertsTour-application-link stalls DO page (a CRITICAL "STRAND" alert, running every 5 min since server boot). The more general tour-outbox reconciler is built, tested, and merged — but nothing in production calls it: no cron, no instrumentation wiring. PM-forward and renewal-escalation resolve delivered-vs-not synchronously, so there's no intermediate "pending" state for them to strand in. | IN PROGRESS | wired: application-link-processor.tsbuilt, unwired: tour-outbox.ts |
| Source-of-income deterministic tripwireA real regex guard exists and blocks voucher/Section-8 language — but only inside one narrow generator (the automated "prospect gone quiet" nudge text). Clara's live chat/SMS/email/voice replies and staff-authored answers both rely on an LLM judge instead, with no deterministic backstop. See also §4.6 fair-housing, which found the live screen exempts residents entirely. | IN PROGRESS | generate-followup-message.ts:215PR #2667 |
| Known-bad SOI conversation pinned as a regressionThe live Willows miss ("No, we don't take section 8" relayed verbatim and taught as policy) is now 11 pinned tests; both fair-housing harness scenarios flipped from expected-fail to must-pass and pass live against the real judge. See §4.6. | DONE | PR #5891reply-release-fair-housing-hold.test.ts |
| Teach-loop review gate for protected categoriesEvery staff decision now runs the fair-housing check before it can move the matter, reach the resident, or get taught as policy; a non-compliant answer holds and asks for a revised one. It routes back to the same staff member, not a separate legal reviewer. See §4.5 and §4.6. | DONE | PR #5891reply-release.ts |
| Trajectory-judge input fixes, replay re-runPart of the scoring stack that went fully live 2026-08-19. | DONE | hardening: #5888 #5906 #5917 #5923 |
| Policy gate armed repair-only on text, one propertyBuilt, but the founder ruled the pre-send gate stays off for now — no arming, no shadow mode. | ON HOLD | decided 2026-08-20 — would-flag review |
| ElevenLabs native test suites turned onThe wrapper and a 3-scenario proof set are merged, but nothing runs them automatically — no CI job, no cron. The only real execution was one manual run against an agent retired since 2026-05-22. The overnight inspection went deeper: the runner able to FAIL was switched off seconds after being enabled and has never run, while the nightly voice check that does run is structurally unable to report failure — details and the fix decision on the eval inspection. | IN PROGRESS | PR #981scripts/elevenlabs/ |
Score store: interim MachineGrade store + Quality DeskLive and iterating at /clara/playground (renamed Quality Desk); 385-label golden set imported to the prod desk. | DONE | #5666 #5906 #5888 #5917 |
| Score store: in-house trend dashboardDecided 2026-08-18 (D1 = in-house): no vendor lens for scores. The store exists; nothing aggregates or charts it over time yet. | NOT STARTED | no trend/chart component in src/components/domain/grading/ |
Cerberus quality engine v0.1.2Consolidated into its own repo (PropFlow-Technologies/cerebrus), vendored into quality-gate-poc/ tag+checksum pinned, synced via scripts/sync-cerebrus.sh. Corpus = 21 golden fixtures. Case-22 (the 2026-08-22 unbacked "you're all set" tour-reschedule incident, below) is upstream in cerebrus, not yet in the vendored corpus. | DONE | #6039 #6072 (spelling) |
CI: Regression Gate — Leasing (required)Runs on every PR. Not yet in branch protection — deliberate burn-in period per founder decision. Ships alongside an advisory gate and an advisory judge lane (claude-sonnet-4-6, subscription-billed). | DONE | #6009 #6045 |
| quality-replay: paired-replay spike + Camellia-scale batch runnerWatch-mode batch runner exercises the real Clara brain at Camellia scale with zero-outbound interception proven (no live sends). | DONE | #6043 #6063 #6071 |
| Quality Desk hardening — first real annotation sessionFrom the founder's first live annotation pass (2026-08-22). Queue is now one newest-first, person-legible list — the ranked inbox is retired; its priority signal was the placeholder rubric (ADR-0128 Amendment 2). Verdict rail: pinpoint saves fixed, handles multi-message turns, reads back the prior verdict, no auto-jump to the next item. First-session annotations verified landing in the GradeVerdict corpus. | DONE | #6106 (queue) #6110 (rail) |
| Voice truthfulness evidence: "you're all set" reproduced deterministicallyOffline eval reproduces the unbacked tour-reschedule completion claim 20/20 with the current tool wording; 0/20 on both honest wordings tried. Post-hoc fake-detector gaps pinned by tests. Key architectural fact surfaced: the voice channel runs inside ElevenLabs' own loop and bypasses the SMS/email outbound safety pipeline entirely. | DONE | #6112 (merged) |
| Rail UX redesign batchRight-click pre-selects the turn; desktop popover (not a mobile drawer); failure-type selector (repair/hold/block); quote+correction note split; blind machine-grade reveal; session meter. | IN PROGRESS | in flight |
| Queue groups by person, not channelFounder rule 2026-08-22: one conversation per person, never split across channels. | IN PROGRESS | in flight |
Case-22 upstream-first landingPath: cerebrus PR #2 → founder merge + v0.1.3 tag → sync PR #6108 into quality-gate-poc/. Domain classification corrected to "other" per the mechanism rule — meaning the REQUIRED leasing gate would not have caught this case; only the advisory lane would have. Relevant to the future gate-promotion decision. | IN PROGRESS | upstream: cerebrus PR #2 (open) sync: #6108 |
| Voice hallucination-guard initiativeFounder-commissioned 2026-08-22. Scope: prior-art + ElevenLabs best-practices research, an options catalog, a throwaway-agent latency prototype, then a decision doc. Constraint: no new hot-path guard without latency evidence; preferred shape is a tool-level check-before-confirm with conflict-explicit tool feedback. Nothing voice-related merges without explicit founder approval. | IN PROGRESS | research + prototype, no PR yet |
| Gate enforcement flip + required-check promotionUnchanged: founder position is "not yet / until it's baked." | ON HOLD | see Regression Gate row above |
| Grading principle ladder Phase 3Distiller / corpus gate / promotion / injection still on 4 unmerged branches (see row above, "Distiller & promotion pipeline"); Phase 3f counter still unwritten. No prompt auto-changes anywhere. | ON HOLD | see "Distiller & promotion pipeline" row |
| Judge calibration report v1 | DONE | scripts/eval/calibrate-judges-subscription.tsPR #5898 |
| Skip-never-green fixed in CIThree separate skip-as-pass bugs (eval sweep, label-only re-runs, skip-passed reviews) each fixed and live. | DONE | #4214 #5622ci.yml label_only guard |
A prompt regression demonstrably cannot merge; every human 👎 lands in the corpus without manual work; zero session-scoped watchers remain.
| Item | Status | Evidence |
|---|---|---|
| PR gates blocking on agent surfacesDecided 2026-08-18 (D3 = blocking): build is authorized. Not wired yet — required checks are still only Build / Type Check / Unit Tests / review; the behavior-eval and red-team suites exist but stay non-blocking by explicit design (cost-capped, 1 sweep/day). | IN PROGRESS | branch protection contexts (live check)clara-behavior-evals.yml |
| Distiller & promotion pipeline (verdict → rule → replay → prompt)The overnight fleet (2026-08-19) found the principle-promotion pipeline FULLY BUILT and green in four PRs deliberately held with explicit "Do not merge" markers since 2026-08-15 (ADR-0128 D5): distiller + versioned principles doc, corpus re-run gate, promotion arm, and prompt injection bound to an eval receipt. Merging them is an open founder decision on the eval inspection page. Still true: the narrower "👎 → auto-drafted regression CASE" lane (ADR-0128 "D8") has no code. Founder 2026-08-20: stays held until agentic quality and testing significantly improve — do not merge yet. | ON HOLD | #5782 #5784 #5785 #5786 |
Corpus-runner wired into CIThe gated replay tool is built (FROZEN/REGRADED comparison, a real --gate flag) but lives only on an open, unmerged PR; not referenced by any workflow. | IN PROGRESS | PR #5784 (open) |
| Native GitHub merge queueBuilt and used in July (4 PRs), then deliberately retired: 66 CI trains for 63 merged PRs, ~25% of the Actions bill, zero genuine cross-PR catches. This is a closed, documented reversal (ADR-0107), not a stalled effort — merges route through the custom script instead. | DROPPED | PR #4303 |
| Durable server-side CI watchersA 15-minute cron sweep pages Slack on a stuck review; auto-merge arms/disarms off native GitHub webhooks. Both run independent of any local session. | DONE | review-verdict-watchdog.ymlauto-merge-all.yml |
| Monthly clustering job v1, scheduledThe script is real and has produced one real result (179 Camellia conversations, August). "Scheduled" is not accurate yet — no cron, no Temporal wiring; human-run only. | IN PROGRESS | PR #5853cluster-conversations-subscription.ts |
| Founder's 100-trace taxonomy passNo artifact or doc found describing this pass. The adjacent golden-set import (385 mined judgements, 132 confirmed live) is real but is verdict-mining from Fede's own history, not a documented taxonomy read-through. | NOT STARTED | adjacent, not the same thing: PR #5857 |
| Voice literal-name lintNo lint or CI check exists. The only related work is a one-time manual scrub of two prompt files, whose own PR body lists remaining occurrences it left unfixed. | NOT STARTED | PR #5557 |
| Unsynced voice agents onto CI syncThe roadmap names two; today's sweep found three live agents with no automated prompt sync — vendor-calling, emergency-relay, and escalation-callback (the last explicitly excluded from even the drift-detector's scan list). A 6-hourly drift check now alerts on staleness for two of the three but fixes none of them. | IN PROGRESS | voice-prompt-drift-check.tsvoice-prompt-drift.yml |
Every allow/deny/repair logs a policy ID + version; a policy change needs no deploy; a protected-category answer cannot lock in unreviewed.
| Item | Status | Evidence |
|---|---|---|
| Policy store v1 — state tier (TX + expansion)No state-level or cross-property tier exists; property knowledge is hard-isolated per building by design. Decided 2026-08-18 (D4 = on-signing): the trigger for building this out is the first out-of-state contract signing, not proactive work now. | NOT STARTED | src/lib/data/types.ts:2529teaching-capture.ts:24-28 |
| Per-policy telemetry (policy ID + version)Two small registries already log which policy fired (a policy ID), but neither logs a version — no policyVersion/ruleVersion field exists anywhere. | NOT STARTED | policy-gate.ts:361-372outbound-safety.ts |
| Jurisdiction review gate on answer-becomes-policyAn explicit 2026-08-13 owner ruling removed an earlier reviewed-gate draft: "a staff reply IS the policy, live immediately." Current protection is provenance-only (who/when/which matter), not a jurisdiction-aware pre-lock-in check. See §4.6 for the same gap on the send side. | NOT STARTED | teaching-capture.ts:9-14 |
| Pricing-data isolation invariant + testsNo matching code or tests anywhere in the repo. | NOT STARTED | repo-wide grep, zero hits |
| Guard chain → declarative pipelineThe pattern is proven twice at small scale (a 2-policy registry, then a 1-policy registry explicitly modeled on the first) — but the bulk of the real guard chain (the hallucination guard, three turnover guards, the same-turn escalation constraint) is still hand-written inside the 9,462-line conversation-manager.ts, none of it migrated. | IN PROGRESS | outbound-safety.ts (#5285)policy-gate.ts (#5815 et al.) |
| Breaker isolation (per-caller-class circuit breakers)Breakers already split by external service (Twilio, SES, DynamoDB…), but the single shared model-call breaker still spans the live tenant conversation loop and 10+ unrelated low-stakes callers (spam classification, portfolio-report narration…) — a burst of failures in any of them can still silence live conversations. | NOT STARTED | src/lib/platform/resilience.ts |
| F5 floor decided and shippedNot covered by today's verification sweep. The doc's own §02 still names this as an open gap (total loop failure produces deliberate silence with no floor except the life-safety lexicon); nothing found today shows a decision or a ship. | NOT STARTED | see §02, F4–F5 — not re-verified this pass |
| Staged prompt rollout on per-property flagsThe flag mechanism itself is shipped and real, but only gates four ON/OFF kill switches — nothing rides it for prompt content or version yet. | NOT STARTED | turn-integrity-flags.ts |
A new firm reaches first-conversation without engineering hand-edits; their ops lead can see their own quality dashboard; the methodology page is public.
| Item | Status | Evidence |
|---|---|---|
| Client-scoped quality dashboard | NOT STARTED | not covered by today's sweep — phase gated behind Phases 1–3 |
| Onboarding wizard closes engineer-required gaps | NOT STARTED | not covered by today's sweep |
| Quality-gated go-live ramp as standard | NOT STARTED | not covered by today's sweep |
| PMS adapter port (read side) + second PMS spike | NOT STARTED | not covered by today's sweep |
| Voice simulation + load testing | NOT STARTED | not covered by today's sweep |
| Published evaluation methodology v1Decided 2026-08-18 (D5 = no): do not publish. Proof-in-public stays an internal asset for now, not a public trust play. | DROPPED | decision D5, §10 |
| Contractual data portability | NOT STARTED | not covered by today's sweep |
| First CS/implementation hire scoped | NOT STARTED | org/people decision, outside repo verification scope |
A new state is a data + review exercise, not an engineering project; the launch checklist is boring.
| Item | Status | Evidence |
|---|---|---|
| State tier populated + legally reviewed per launch stateDecided 2026-08-18 (D4 = on-signing): trigger is the first out-of-state contract signing. | NOT STARTED | see D4, §10 — not independently re-verified this pass |
| Per-state disclosure behavior wired | NOT STARTED | not covered by today's sweep — phase gated behind Phases 1–4 |
| Dual-tree module resolution retired (F1) | NOT STARTED | not covered by today's sweep |
| Loop decomposition continues behind the regression suite | NOT STARTED | not covered by today's sweep |
| Temporal-wrapped conversation turns where duty-of-delivery demands it | NOT STARTED | not covered by today's sweep |
The three-month evidence base is blunt about our habits, and habits ship architecture. We commit to four changes, stated in the same register as the doctrine:
Founder decisions gating the roadmap. Answers save for everyone on this page; recommendation marked.
Decisions H1–H4 below come from the honesty-layer redesign (§4.5, merged from the 2026-08-16 inspection) and are still open.
Q1 below comes from the 2026-08-21/22 finding that the daily post-hoc prod-conversation judge is paused; full writeup on the Quality System Plan.
conversation-review-daily, the daily job that reads every prod conversation after the fact and grades it, has been paused since ~Aug 16 — its planned cutover to sole reviewer never happened, and the team's planned replacement (a live #apartment-camellia activity feed) is an events feed, not a judge. Since ~Aug 16, nothing reviews a finished conversation after the fact. OPEN — awaiting the founders. Detail and the fuller three-way decision: q-posthoc-next on the Quality System Plan.
Evidence base (internal). Three-month mining pass over 402 working sessions and 4,393 merged PRs (1,732 fix-classified, 28 reverts); five incidents with written root causes, including the June renewal-migration dropped-decline and the August honesty-guard false-block revert; a same-day audit of every voucher/assistance conversation in 90 days of traffic (3 real questions: 2 routed correctly, 1 answered directly — the trigger appears to be an already-fixed context bug, but the fair-housing check scored the violating reply "Compliant," which is why the deterministic tripwire is Phase 1). Full fleet reports and per-finding session citations: ~/Documents/propflow-architecture-scan-2026-08/ (internal archive).
Current measured baselines (as of 2026-08-18). Voice simulated suite: 278/317 green, failures concentrated in vendor-outbound (20/35) — the one agent without CI sync; renewal gauntlet: 90.9% tool correctness, 97.7% no-invented-promises, 100% payment-safety, 72.7% pacing (44 scenarios); intent golden set 84/85 with production-trace drift 0/30; unsupported-claim rate ~19/month on real traffic (SMS catcher precision 5:1, email 4:3, voice 10:28 pre-fix). The flagship behavior sweep has completed zero runs in its last 40 attempts — Exhibit A for P7.
LangGraph proof-of-concept — CLOSED 2026-08-18, five learnings. The experiment answered its question and its PR is closed unmerged (branch kept as a reference implementation; eval artifacts in the off-repo run archive). Final measured result: 262 real leasing conversations replayed head-to-head, 60% of POC turns same-intent-or-better than production (15% strictly better), 90.1% grounded, 100% fair-housing-safe; five capability scenarios at 25/25 hardened checks. What we keep: (1) no framework rewrite — LangGraph's durability features duplicate Temporal, and the loop itself was never the weak layer; (2) two patterns adopted — the declarative guard pipeline (shipped as the policy gate) and suspend-until-a-human-answers escalation with the answer persisted as property-scoped policy, never re-asked; (3) the replay surfaced a real production fair-housing miss (a voucher question answered directly with dollar math, scored "Compliant" by the live single-pair check) — the finding behind the Phase-1 SOI tripwire and history-aware post-hoc scoring; (4) evidence layers lie until attacked — five adversarial review rounds on the POC's own harness removed fabricated-verdict paths, and the honest harness scored lower (49%→43% raw match), which is the proof program applied to itself; (5) escalation scope must be declared policy, not prompt-prose accident — production and the POC drew the answer-vs-route line differently and neither had decided on purpose.
External research inputs. "Production Reliability, Guardrails & Testing for Agentic AI" (2026 synthesis: converged guardrail architecture, trajectory-aware judging, tool-boundary validation, eval-driven development; vendor figures treated as directional, not audited) · industry survey of Decagon, Sierra, Intercom Fin, Harvey, Salient, Gradient Labs, EliseAI · enforcement record: Air Canada, Harbor Group/PERQ consent decree, Mata v. Avianca, DoNotPay FTC settlement, DOJ v. RealPage, SafeRent, HUD 2024 guidance · evaluation practice literature (error-analysis-first taxonomy building, judge calibration, Clio-pattern clustering) · platform landscape as of Aug 2026 (promptfoo → OpenAI; Langfuse → ClickHouse, license unchanged; Arize → Dynatrace pending; Helicone/Traceloop effectively discontinued; Humanloop absorbed into Anthropic).
Companion pages. Honesty layer deep inspection (distilled into §4.5) · fair-housing screening architecture (distilled into §4.6) · honesty gate data sheet · escalation decisions · the eval system, deep inspection (54-suite promptfoo inventory, the Sonnet 5 story re-verified, six open decisions) · the quality system plan (one measurement pipeline, CI regression feedback, the outbound checkpoint rebuild — the active plan superseding §08 Phase 2 above) · the Eval & Testing Roadmap (the living gap-by-gap tracker for both of the above) · vision gap analysis (superseded).