Should PropFlow bring in an open-source project or a commercial platform for the after-call system (what happened on a call, tracking Clara's promises, getting a human involved) — or keep building it thin on Temporal? Six areas researched independently against real, current products and pricing.
2026-08-26 · commissioned by Fede · research only, no purchase made · every pricing figure below is sourced or marked "not public" — nothing invented
Proposed — pending FedeBottom line: build thin on Temporal, adopt two free/already-paid-for pieces, buy nothing. Six independent research passes — durable execution, call-outcome evidence, promise/obligation tracking, human-escalation paging, eval tooling, and agent frameworks — all landed on the same shape: every commercial product in this space is priced and designed for a human team working a queue, and PropFlow's actual shape is one AI agent generating machine volume that no seat-priced tool is built to carry economically. The bugs that prompted this question (a false page to a property manager, a placeholder text sent instead of real content) were mistakes in PropFlow's own application code, not evidence the code shouldn't exist — and no vendor researched can do the one thing that actually matters: know PropFlow's own residents, leases, and call data well enough to tell what really happened.
Three pieces, all already named in this week's own incident write-ups:
This is measured, not guessed, from the actual codebase: the promise ledger is about 3,800 lines of code, and the human-notification ("matter") lane is about 5,800 lines — roughly 9,600 lines of homegrown infrastructure today. That size is what prompted the question this document answers.
| Area | Verdict | Why, in one line |
|---|---|---|
| 1. Durable execution waiting on real events, durably | Keep Temporal | The bug is a hand-rolled short poll loop racing a variable-length real event — an application mistake, not a platform gap. Temporal already has the exact pattern (signals + durable timers), documented by name. |
| 2. Call-outcome evidence what really happened on a call | Build / reuse what's paid for | No vendor sees Twilio's own call-leg signaling — every one would add a second, disconnected opinion to reconcile. The fix is switching from polling to ElevenLabs' own webhook, already in the stack. |
| 3. Obligation tracking promise ledger's residual scope | Keep DynamoDB + Temporal | Every ticketing/CRM/SLA product prices per human seat and still can't verify a specific spoken promise — that judgment has no vendor, at any price. |
| 4. Human-notification lane one deduped door for a human | Keep, extend the matter lane | incident.io/PagerDuty are built for engineers on-call; Front/Zendesk/Intercom are built for humans working a queue — the exact shape PropFlow has a standing rule against. None model a resident + lease + "answer becomes policy." |
| 5. Eval / guardrail harness catching bad messages before they send | Already built — Promptfoo in production | Braintrust, LangSmith, Patronus all require metered API billing PropFlow's own rules forbid — enforced in code by the repo's own metered-spend-guard. Promptfoo isn't a candidate to pilot — it's already running 58 eval configs across 5 CI workflows, wrapped by ~42 subscription-runner scripts. Correction, Aug 26: the first version of this page missed this and proposed piloting a tool already in production — see section 5. |
| 6. Agent framework LangGraph, CrewAI, etc. | Adopt one narrow piece | Every framework's "durability" story duplicates or fights Temporal (one team's public case study shows them ripping LangGraph out in favor of Temporal). Adopt only Anthropic's own tool-calling loop, dropped inside a Temporal step. |
PropFlow already runs Temporal Cloud for everything durable: renewal workflows, escalations, the promise ledger. The bug that raised this question: a checker waits for a phone transfer's outcome on a fixed ~2.5-minute clock (six tries, 25 seconds apart) — but the real answer (the call recording finishing) can take much longer. On one real call it paged a property manager six minutes before a human had already, correctly, handled the caller. That's a short clock racing a real-world event with no fixed length.
Imagine setting a 2-minute timer to find out if someone answered the phone, then giving up and telling the boss "nobody answered" — even though the call is still going, 9 minutes in, going fine. The fix isn't a new tool. It's: don't set a short timer for something that finishes whenever it finishes — wait for the actual "the call is over" signal, with a much longer backup timer in case that signal never comes.
| Option | Has this pattern? | Price (sourced) | Verdict |
|---|---|---|---|
| Temporal (current) | Yes — signals + durable timers, documented as the fix for exactly this shape | Essentials: greater of $100/mo or 5% of usage, 1M actions included, $50/M over. Business: $500/mo min, 2.5M actions included. Storage $0.042/GB-hr active. (temporal.io/pricing) | Keep — fix the code |
| Restate | Yes — "awakeables" (a durable promise resumed by an outside callback) | Free tier 50k actions/mo; paid tier pricing not public. Server license is source-available (BUSL-1.1), not fully open like Temporal. | Capable, but switching platforms to fix one checker is disproportionate |
| Inngest | Yes — step.waitForEvent(), times out to a known value | Free 50k executions/mo; Pro from $99/mo, 1M included. Excellent fit for Vercel/Next.js teams starting fresh. (inngest.com/pricing) | Good default for a greenfield pick — not a reason to run two orchestrators |
| AWS Step Functions | Yes — task tokens + heartbeats, waits up to a year | $0.000025/state transition, 4,000 free/mo. Same vendor as PropFlow's existing Lambda/SQS. (aws.amazon.com/step-functions/pricing) | Cheap, but declarative JSON authoring is a worse fit than Temporal's code-first workflows; would mean running two orchestrators |
The company's own principle: "we record every call, we should know this." The evidence already exists (Twilio's own call record, the recording) — the problem is reading it correctly, not acquiring more of it. One real bug found in this research: the stored outcome for one call said "connected" while Twilio's own log showed a 4-second no-answer.
PropFlow's phone system (Twilio) and its voice AI (ElevenLabs) already both know what happened on a call — one knows "did the phone ring and connect," the other knows "what did Clara say and hear." A call-transcription company can only ever tell you the second part, and usually less reliably, because it never sees the phone signal at all. Buying one doesn't fill the gap; it adds a second guess to check against the two answers already on hand.
| Product | What it actually offers | Price (sourced) | Data exposure | Verdict |
|---|---|---|---|---|
| ElevenLabs' own post-call webhook | Already in the stack. Pushes call-end data (why the call ended, whether it succeeded, every tool Clara used and its result) the moment the call finishes — no need to poll for it | $0 — already paid for as part of existing voice spend (elevenlabs.io docs) | None new — same vendor already in the stack | Adopt — this is the fix |
| Twilio Voice Intelligence (Custom Operators) | A configurable "read the transcript, answer a yes/no" layer on top of Twilio's own recordings — could replace some hand-built transcript checks | $0.002/1,000 input characters + $0.018/1,000 output characters (twilio.com pricing) | Same vendor already in the stack, but confirm which model actually reads the transcript before using it on sensitive topics | Worth a narrow, optional pilot |
| AssemblyAI / Deepgram | Pure transcription — no visibility into whether a call was even answered; Deepgram's own reference build has the AI guess what happened rather than checking the phone record | AssemblyAI ~$0.15–0.21/hr; Deepgram ~$0.0043–0.0078/min (assemblyai.com, deepgram.com) | New vendor sees recorded resident conversations, including any disability/income/family-status content | Don't adopt — doesn't reach the actual bug |
| Gong / Chorus-class | Built for B2B sales teams tracking deals and quotas — no property-management concept at all | Not public; third-party estimates ~$1,300–2,000/seat/year + $5,000+ platform fee, 15-seat minimum (unverified estimate) | New vendor, wrong category, per-seat pricing that doesn't match "one AI agent," not humans | Wrong category |
One writer, not two: the ElevenLabs webhook is an input to the Call Evidence Resolver, not a second, parallel place that decides call outcome. The resolver stays the one place that turns evidence into a verdict — it just starts listening for the webhook's push instead of polling for it, and still cross-checks against Twilio's own call-leg record before deciding anything. Two writers of "what happened" is the exact failure mode section 2 opened with; this change must not recreate it under a different vendor.
Fede already ruled on the direction here: shrink what counts as a "promise" that needs tracking at all — most outcomes should be stated as already-true facts backed by proof (a work order number, a confirmed tour), not tracked as an open commitment. What's left — a genuinely open human follow-up — still needs an owner, a due time, and a way to mark it done.
| Category | Sharpest finding | Price (sourced) | Verdict |
|---|---|---|---|
| Salesforce (CRM task engine) | Salesforce's automatic escalation feature only exists for one specific object (Cases), and only one such rule can be active company-wide. The everyday "task with a due date," the part that actually matches "Clara owes a follow-up," has no automatic escalation at all — you'd build it yourself, in Salesforce, at Salesforce prices | Enterprise $175/user/mo → Agentforce 1 Service $550/user/mo; ~$60k/year realistic deployment before setup (source) | Don't adopt — relocates the same build work, at a much higher price |
| Zendesk / Front / Intercom (SLA features) | SLA targets are generic ("first reply within X hours") — never a custom rule like "did the automation's own receipt show up" | Zendesk SLA needs Suite Growth+; Intercom SLA gated to its priciest $132/seat/mo tier (intercom.com) | Don't adopt |
| Gong / Dialpad Ai ("commitment extraction") | These find commitment-shaped sentences in a transcript and drop them into a CRM as a note — they don't own the "is it done yet, escalate if not" part at all | Not public; third-party estimate $5,000–15,000+/yr + $108–250/user/mo (unverified) | Doesn't reach the problem |
| Emerging AI-agent-native tools | Honest finding: nothing purpose-built and mature exists yet for "an AI agent's own promises, tracked at machine volume." The one product marketing "commitment tracking" (Claryti) is built for a person's own ~23 commitments a week, not thousands generated by software | $15/seat/month | Not applicable at PropFlow's volume |
Today, 13 separate places in the code can page a property manager — a stuck renewal, an unanswered transfer, a maintenance approval — and none of them check whether a human has already been told about this exact issue. The plan already agreed: collapse all 13 into one "matter" per resident/issue, with a check before opening a new one, and a rule that once a human answers a question, that answer becomes standing policy and Clara never asks again. Standing company rule: never build a list of things for a property manager to work through — everything is automatic by default, and a human only ever sees one card at a time.
The tools built to page an on-call engineer (PagerDuty, incident.io) assume the reader is a software engineer fixing a system outage at 2am — wrong audience, wrong urgency, no idea what a "resident" or a "lease" is. The tools built for a shared support inbox (Zendesk, Front, Intercom) assume a person is sitting there working through a list of open tickets — exactly the kind of "queue for a human to work" PropFlow has a standing rule against. Neither kind knows PropFlow's own rule that "once you tell Clara the answer once, she should never ask again."
| Product | What it actually does well | Price (sourced) | Why it's the wrong shape |
|---|---|---|---|
| incident.io | Its de-duplication idea is genuinely worth copying: one "key" per issue — while that key is open, no second alert fires | Free ≤5 users; Team $19/user/mo; Pro $25–45/user/mo (source) | Built for engineers watching Slack during an outage — no resident/lease model, no "answer becomes policy" |
| PagerDuty / Opsgenie | On-call scheduling and escalation for system faults | PagerDuty Free ≤5, Pro ~$21–25/responder/mo, Business ~$41–49/responder/mo (source) | Opsgenie is being shut down (support ends April 2027) — a non-starter regardless of fit |
| Front / Zendesk / Intercom | Merging duplicate tickets after the fact; canned replies for common answers | Front $25–105/seat/mo; Zendesk realistic total $165–265/agent/mo; Intercom $29–132/seat/mo, AI resolutions billed separately at $0.99 each with a 50-resolution minimum (sources: Front, Zendesk, Intercom) | Merging happens only after a duplicate already exists — not a check before opening one; built for a human working a queue, the shape PropFlow avoids by rule |
| HumanLayer / LangGraph interrupts / AG-UI | A way for one AI task to pause and ask a person a question mid-task | HumanLayer pricing inconsistent across sources — not independently confirmed | These are single-question tools, not a ticket/matter object — no dedup, no policy memory. AG-UI is a wire format for streaming an agent's state to a screen, not a hosted product at all |
Direct, relevant history: PropFlow adopted Langfuse (a tracing/observability product) earlier in 2026, then removed it in July because it added more day-to-day complexity than it saved — replaced by an internal store the team fully controls. There's also a hard billing rule: Claude usage for evals must run on the team's paid subscription, never a metered API key, after a past mistake burned real money the wrong way. That rule disqualifies most of the SaaS options in this category by itself — but it's worth noting PropFlow already solved it for the one open-source tool it actually uses, well before this research pass.
| Product | Works with the subscription billing rule? | Price (sourced) | Status at PropFlow |
|---|---|---|---|
| Promptfoo (self-hosted) | Yes — Promptfoo itself added real support in April 2026 for authenticating with the same login used for Claude subscriptions (source); PropFlow's own repo additionally wraps every run in a subscription-runner script so a metered-key path is never reachable, per the repo's metered-spend-guard (a prior PR adding an API-key fallback was rejected) | Free, open-source, self-hosted. Acquired by OpenAI in March 2026 — kept open-source, but now roadmapped inside a direct competitor to Anthropic (source) | Already adopted — 58 configs, 5 CI workflows, ~42 subscription-runner wrapper scripts, version pinned at 0.121.20 |
| Braintrust | No — expects a raw API key pasted in directly | Free Starter; Pro $249/mo (source) | Fails the billing rule — same shape as Langfuse |
| LangSmith | No | Free Developer; Plus $39/seat/mo (source) | Fails the billing rule |
| Patronus AI | N/A — uses its own proprietary judge model, not Claude, billed per call | Base $25/mo + $10–20 per 1,000 evaluator calls (source) | Replaces Claude's judgment with a black box |
| OpenAI Evals | No — built around OpenAI's own models | Free, but the hosted product is being fully shut down November 30, 2026 | Wrong model family, and dying |
| DeepEval / inspect-ai | No, as shipped | Both free/open-source libraries | Fails the billing rule as-is; inspect-ai is built for safety research, not a deploy gate |
None of these seven know PropFlow's own incident history or promise-detection rules — that has to be built in-house either way, which is what Cerberus already is. None of them close the voice-channel gap either: real voice testing (audio conditions, interruptions, latency) is a different, separate product category not covered by this pass. Promptfoo, the one already in production, doesn't change either of those two facts — it's a runner/config layer, not a replacement for Cerberus's domain-specific judgment.
PropFlow calls Claude directly today, with hand-written code deciding which tool to use next. No name-brand "agent framework" (LangGraph, CrewAI, and similar) is in use. Company principle: Clara is one brain across leasing, maintenance, renewals, and collections — not a patchwork of separate bots — so any framework has to support one coherent agent, not encourage splitting it up.
| Framework | Adopt? | Why |
|---|---|---|
| LangGraph | No | Its own comparison page against Temporal admits its save-points only preserve data — recovering from a crash is manual, not automatic like Temporal. The one real production story published (a Fortune 500 manufacturer, via Temporal's own case study) is a team that removed LangGraph in favor of Temporal, deleting "thousands of lines" of their own retry code in the process (source) |
| CrewAI | No | Built around splitting work across separate role-playing agents — the opposite of "one brain." Four security vulnerabilities disclosed in 2026, serious enough that CrewAI removed one of its own tools entirely (source). $25/mo + $0.50 per run beyond the free tier (source) |
| Microsoft Agent Framework / AutoGen | No | Its durability story runs on Azure — a brand-new cloud dependency PropFlow doesn't have, duplicating what Temporal already does |
| OpenAI Agents SDK | No | Wrong model provider — PropFlow is Claude-native. OpenAI's older Assistants API is itself being retired August 26, 2026 |
| Claude Agent SDK (Anthropic's own) | Yes, narrowly | Anthropic's own maintained version of "call Claude, check if it wants to use a tool, run it, repeat" — makes no competing claim about durability or multi-day waits, so it drops cleanly inside a Temporal step and simply replaces hand-written boilerplate with a version Anthropic maintains |
Every figure below is a labeled estimate, not a quote — built from the sourced per-unit prices above and a rough guess at call volume (~10,000 calls/month across leasing, maintenance and renewals, ~15% involving a transfer). Nothing here should be read as a real invoice.
| Area | If bought (rough) | If built (the current path) |
|---|---|---|
| Durable execution | N/A — staying on Temporal either way | Already-paid Temporal Cloud spend; the fix likely lowers the bill (fewer repeated checks billed as actions) |
| Call-outcome evidence | Twilio Custom Operators only, narrowly: roughly $10–100/month at 10,000 calls (rough estimate, depends on how much transcript text gets classified) | $0 incremental — reusing the ElevenLabs webhook and Twilio call data already paid for |
| Obligation tracking | No viable option found at any price — every product prices per human seat and still needs PropFlow's own logic built on top | $0 incremental — engineering time only, on infrastructure already running |
| Human-notification lane | incident.io Pro ~$25–45/user/mo, or Zendesk/Front/Intercom $25–265+/seat/mo — all the wrong shape | $0 incremental — extending ~5,800 lines already built |
| Eval / guardrail harness | Braintrust $249/mo, LangSmith $39/seat/mo, Patronus $25/mo + per-call fees — all fail the subscription-billing rule | Cerberus is sunk cost, already its own repo; Promptfoo is already running at $0 incremental, wired through ~42 subscription-runner scripts — not a future pilot, already the current state |
| Agent framework | CrewAI $25/mo + $0.50/run — real usage-based cost at this volume, and not recommended regardless of price | Claude Agent SDK: no separate fee, same billing as today |
Directionally: in four of the six areas, buying is actually more expensive than building, because every vendor researched prices for a growing human team, not a system where one AI agent generates the volume. The two areas with a real "yes, adopt" are both free or already paid for.
Build thin on Temporal. Adopt two narrow, free, zero-new-vendor pieces. Buy nothing. This isn't a conclusion reached first and defended after — six separately researched areas converged on it independently, four with no close call at all. The reason repeats because it's structural, not a coincidence: every commercial option in this space is priced and built for a human team working a queue, and PropFlow's real shape is one AI agent generating machine-speed volume that a seat-priced tool was never built to carry.
The roughly 9,600 lines of in-house code that raised this question isn't, on this evidence, proof of over-building. It's the visible cost of judgment calls no vendor can make — what counts as a kept promise, what counts as "someone already knows," what really happened on a call — because all three require knowing PropFlow's own residents, leases, and call data. The bugs that triggered this whole question were mistakes inside that code, not reasons the code shouldn't exist. Fixing them is smaller, cheaper, and safer than replacing any of it with a vendor.
(a) Recommended — this is the answer. Confirm building in-house across all five core pieces, adopting only the two zero-new-vendor pieces already described above (the ElevenLabs webhook switch, Anthropic's tool-calling helper). No new vendor contract, no new resident data leaving the company, no new per-seat cost.
(b) Before fully committing, run one small, real pilot of a single named product (e.g., Twilio's transcript-classification feature, for that one piece only) with a hard stop date, to sanity-check this research against a week of real production data.
(c) Pause further building on the after-call system entirely, until a smaller version of just the highest-severity piece (the transfer-outcome timing bug) is done and proven.
The choice was: trust this research and keep building the way the team already is, or spend a week trying one outside tool first. Fede picked the first one — full speed ahead on Temporal, nothing bought, nothing piloted.
(a) Recommended. Fix it now — replace the short repeating check with a real "tell me the moment it's actually over" signal, backed up by a much longer safety timer. This is the direct, permanent fix for the false-page incident, costs nothing new, and Temporal (the tool already in use) documents this exact fix by name.
(b) Widen the current short timer as a stopgap (say, 5–10 minutes) without changing how it waits.
(c) Leave it as-is and keep manually catching cases like this as they come up.
This is the one clearly-broken thing in the whole document — a system gives up and pages a person before it could possibly know the real answer. It's already fixed: option (a) is what shipped, and the second, smaller piece (making sure a data-capture failure tells an engineer, not a property manager) is being reviewed now.
(a) Recommended, narrowly. (Original text, based on the incorrect premise that
Promptfoo was not yet in use.) Pilot Promptfoo self-hosted, for a sprint or two, strictly
as testing convenience around the existing harnesses — but only wired through
PropFlow's existing subscription-powered eval runner
(scripts/lib/subscription-eval-runner.ts), never through Promptfoo's own default
API-key path. Never the place trace data or the incident history actually lives. Drop it
without ceremony if it doesn't clearly save time, the same way Langfuse was handled.
(b) Skip it entirely, citing the rule — OpenAI bought Promptfoo in March 2026, and a Claude-focused team may not want its testing tool's future roadmap sitting inside a direct competitor to Anthropic, on top of the extra work of routing it through the subscription runner at all.
(c) Stay entirely on Cerberus and the existing test harnesses; only look at outside tools again if one specific, clearly-named gap (like proper voice-call testing) becomes urgent enough to justify a dedicated search.
This whole question was built on a mistake: the research treated "should we try Promptfoo" as an open question when the team had already answered it months ago — it's running 58 test configs across 5 automated pipelines today. There was nothing here for Fede to decide; there was something for the research to get right the first time, and it didn't.
Full working notes, every vendor considered, and the complete source list: see the research file alongside this page. Sources are linked inline throughout — nothing above is an invented number.