The fire alarm nobody wired
welcome to the debate. When you walk into a newly constructed high-rise building, you don't really think about the fire alarms. Right, you just assume they work. Exactly. You see the little plastic disc on the ceiling, you see the sprinkler head, and your brain just automatically assumes safety. But what if you looked behind the drywall, and you realize that the fire alarm didn't actually have any wires connecting it to the building's power grid? Oh man, that's a terrifying thought. Right, so today we are looking at a situation very much like that fake plastic fire alarm. We're analyzing prop flows internal deep inspection of their AI evaluation system for their property management agent Clara.
That's right, and for those who might not know, Clara is an AI that handles real-world maintenance requests, lease renewals, and tenant communications. Yeah, she's talking to real people. Right, and after five generations of rebuilding this evaluation system, the internal report we're looking at is sparked just a massive debate. It really has. So the core question we are exploring today is this. Is prop flows current e-vow infrastructure just a fundamentally broken system providing a dangerous illusion of safety or is it a genuinely capable set of parts that merely requires final integration? I take the position that this system is structurally flawed.
The two positions
I mean, it has critical blind spots and flawed measurement protocols that render the entire apparatus completely toothless. And I take the opposing view here. I argue that the foundational components of prop flows e-vow system are actually incredibly robust, and the current issues are merely mechanical, basically wiring problems awaiting final approval to be fixed. I see why you think that, but let me give you a different perspective. A safety net that is structurally incapable of catching you is not just a technical oversight. It is a failure of the entire system's architecture. Okay. The e-vow system is broken because it cannot fulfill its primary purpose, which is blocking bad code from shipping.
The safety net that runs one time in two hundred
Right, blocking regressions. Exactly. And the evidence from this internal deep inspection is staggering. Let's look at the broadest safety net check they have. Because of a daily cost cap, it only executes one in roughly 200 attempts. Right, but it's a resource constraint. Yes, but the logic inside the check itself actually works. But if it doesn't run, it doesn't work. I mean, that means 199 times out of 200, the system just skips the check, and a skip registers as a green success. Well, and it gets worse when you look at the voice surface, Clara takes inbound phone calls from tenants. Yes, she does phone in text. Right, and this voice surface has two dedicated safety nets.
One was switched off at the platform level, 32 seconds after it was created. Wow. Yeah, 32 seconds, and it has run zero times since. The other one runs every night, but it's hard wired with a soft flag, which is a standard practice during early development to be fair. Sure, but for listeners who might not be deep into software engineering, a soft flag means the test is allowed to fail without stopping the assembly line. Right, it doesn't break the build. Exactly. The pipeline will always report a green success to the system, regardless of what the AI actually did. And during one real run, this checks successfully identified 14 genuine serious problems.
What the fourteen failures actually were
14 14 Clara was confirming tour bookings after the booking had failed in the database. Oh, no. She was booking tours without the caller ever agreeing to them. She was offering time slots on days the property was completely closed. Yeah, that's bad. The test found all of this, and the system still reported success. You cannot call an infrastructure capable when its core design actively suppresses failure. I come at it from a different way. You're focusing entirely on the fact that the cables aren't plugged into the main break or yet, and you're ignoring the absolute marvel of the engine they've just built. An engine that ignores 14 failures.
The case for the defence: six iterations in
But this system is a textbook success story of iterative development. The founders noted they've had five different iterations since March, and this six one is genuinely solid. I'm not sure I call 14 ignored failures solid, but look at the components that now exist because of that build get burned rebuild cycle. They have a 385 case hand annotated golden data set, which sounds impressive, but what does that actually mean in practice? Well, it means they have a massive spreadsheet of past scenarios where humans perfect verified answer sits right next to the machine's answer. When Clara is updated, her new answers are checked against that golden standard.
Pinning the judge
Right. If she deviates, you instantly know you have a regression and to run that check, they have a brand new calibration harness with a pinned judge model. Let's pause on pin judge for a second. We're talking about an AI model grading another AI model, right? Exactly. You have a judge AI grading Clara's work, but AI models can suffer from model drift. If the AI company behind the judge silently updates their system, the judge might suddenly become more lenient or more strict. Oh, I see. Yeah. So a prompt that passed yesterday fails today, not because Clara changed, but because the judge drifted. Proplo fixed this by pinning the judge.
They hard coded it to a specific frozen version of the model. Okay, that makes sense. And when they tested it, this pinned fair housing judge got 85 out of 85 clean examples exactly correct. It is highly accurate and immune to drift. I can see that pinning the judge is smart engineering, but none of that stops a bad merge from going live because they haven't flipped the switch yet. The infrastructure to turn a thumbs down from that judge into an automatic permanent fix and Clara's prompt is entirely built, but it's not active. It distills the failure into a rule, replays it against real history to prove it works, and injects it into the prompt.
Four green pull requests, sitting unmerged
That entire pipeline is split across four finished green reviewed pull requests. Pull requests? So proposed co-changes that haven't been accepted yet. Right. They are sitting in the pipeline fully functional, just waiting. The system isn't structurally broken. It is merely waiting for a human to click merge to plug those cables in. So wiring problem. Well, let's look at a real world scenario to see if the parts are as functional as you claim. Let's talk about the specific catalyst for this whole deep inspection. The Sonnet 5 was worse narrative. All right. Yeah. In early August, an engineer ran a massive 48 sweet comparison and concluded that the new Sonnet 5 model was performing worse than the older 4.6 model.
The model comparison, and its procedural flaws
I argue that this comparison proves the Evau system is highly unreliable. The test certainly had procedural flaws. I'll give you that. Procedural flaws is putting it mildly. First, this massive test was run from an uncommitted scratch folder, meaning it wasn't even run on the official pristine code base. It was run on some developer's messy desk effectively. It never tested the model actually live in production, which is a mistake, definitely. But more importantly, the test itself was fundamentally biased due to a token budget bug. The reply budget issue from July. Yes, that reappeared. Let me use an analogy here. Testing Sonnet 5 with a tiny reply budget so it runs out of room mid-thought is like failing a race car because you only gave it half a tank of gas, not because it's inherently slow.
Chain of thought needs a scratch folder
It's a fair analogy, but let's explain the mechanics of that. The newer clawed five line models use a technique called chain of thought. Right. They essentially need a scratch pad. They output hidden text to reason through a problem before they give you the final answer. Right. So they need a larger token budget, a higher word count limit to do that thinking. The Eval test starved them a space. Yeah. They literally got cut off mid-thought and the Eval system marked it as a capability failure of the AI. That's a flawed measurement. I agree the budget was too tight. And it gets worse. The Eval runners silently dropped the determinism setting for Sonnet 5 because the API rejected it.
Temperature, determinism, and what zero does not guarantee
The temperature setting? Yes. For our listeners, temperature or determinism controls how creative the AI is allowed to be. Exactly. If you set it to zero, the AI should give you the exact same math like output every single time. It's crucial for establishing a baseline and testing. Right. You need that consistency. But the test dropped the setting. So the older 4.6 model got to answer the exact same way every time. While Sonnet 5 was sampling randomly, just hallucinating different responses. It wasn't a fair fight. The results were just measurement noise, masquerading as a genuine signal. I'm sorry, but I just don't buy that. Let me tell you why.
The regressions that survived the bad test
Okay. You are entirely correct about the messy scaffolding. The scratch folder, the token budget, the temperature settings, those were known bugs. But despite all of that messy scaffolding, the core evaluation mechanism still succeeded in surfacing absolute truth. How can you possibly trust the truth of a rigged test? Because of the adversarial recheck, when the 20-agent overnight fleet re-verified the raw saved outputs, not just the automated write-ups, but looking at what the model actually did, they found that the test caught real regressions. Wait, real ones? Yes. We're talking about six to seven distinct confirmed behavioral regressions that survive all scrutiny.
This wasn't just noise. Real regressions from Sonnet 5? Like what? Sonnet 5 was changing real account state meaning actual financial charges and renewal terms on turns where the human tenant merely asked to plain question. Are you serious? Yes. It wasn't supposed to take any action yet, but it did. That's a massive failure. It gets worse. It leaked a tenant's military deployment details into an internal note. Wait, just from a plain question? Yep. It just blurted out sensitive military deployment data? Exactly. It just appended it to an internal system note completely unnecessarily. And it didn't stop there. It broke a call transfer, so the caller was sent to a dead end instead of a property manager.
Acting without speaking
Oh, wow. And across six different suites, it exhibited a recurring pattern of taking actions with zero words spoken out loud. It just fired a tool in the software and hung up the phone in silence. That is so creepy. Right. So finding six to seven real critical behavioral regressions proves that the value suites themselves are highly effective. The harness was flawed, yes. But the mechanism caught very real dangers. OK, so I'll grant you that the mechanism spotted the Sonic 5 regressions. It saw the military data leak. It saw the silent hangups. But that brings up an even scarier question. It saw the problem and then what did it do?
What happens when a check does find something
Well, it logged it. Exactly. Nothing. It did nothing. And this brings us directly to the system's architecture. What happens when these robust parts actually spot a problem? The system is entirely advisory. Today, yes. Today, only ordinary co-checks, like your basic build types or unit tests, can actually block a change from merging into production. The AIE valve just prints a warning. A very detailed warning. But what terrifies me most aren't even the bugs. It's the structural blank spots in the coverage map. You're referring to the ungraded surfaces. Yes. Let's look at the two highest-stakes decisions Clara can make in this entire ecosystem.
Maintenance triage: the graded surface
OK. Number one, maintenance triage and routing did Clara accurately price and route a broken pipe versus a burnt-out light bulb? Right, which is huge for a property manager. Huge. Number two, renewal acceptor decline decisions. Both of these massive, high-stakes surfaces have a grading slot reserved in the system, but there are zero live inputs feeding it. Well, I have to admit, I'm a bit fuzzy on how this even happens. How can you have a grading system that just accepts zero inputs and calls it a day? What is it actually grading? It's grading the conversational wrapper. The wrapper? Yeah, the schema is prepared for the decision data, but right now the system is only looking at the dialogue.
The judgment the roll-up will not make
It will tell you if Clara's tone was polite or if the conversation resolved smoothly, but the actual Boolean decision of, you know, was this the right vendor to call? Doesn't have the historical data wired in yet to make a judgment. So they shipped a real production fix to make sure Clara flags genuine counteroffers correctly on lease renewals, but they never updated the test to check if she actually did it. Not the test date itself, no. The test only checks of some escalation fired, not the right one. As one engineer put it, you are grading the grammar of a bomb threat. That's a dramatic way to put it, but it's true. It doesn't matter if the tone is polite, if she's deploying a plumber to fix a Wi-Fi router.
But again, this is explicitly a wiring problem, not a missing capability. The empty placeholder means the data schema is already prepared to receive the judgment. It's still empty though. But the new scoring system, which shipped just 24 to 36 hours before this deep inspection report was generated, is phenomenal. It grades every text message and every finished voice call twice. Twice? Yeah. Once per message as it settles to catch immediate errors and once after the whole conversation goes quiet to catch those subtle failures where the replies are technically fine, but the conversation just goes in circles. But a detailed scorecard doesn't stop the bad AI behavior from going live.
An autopsy is not a brake
It's just a highly detailed autopsy report. It stops it if you merge the pull requests. We keep coming back to this. In software development, a continuous integration pipeline, a CI pipeline, is basically an automated assembly line. All right. Code gets proposed. It gets tested. And if it passes, it merges into the real product. The pipeline to enforce these fixes and block bad code exists right now in those four untouched PRs we discussed. Okay. The distillation of the rule, the historical replay, the prompt injection, it's all there. So why aren't they merged? The report tells us directly. FedE, one of the founders, decided on August 20th to hold those PRs.
The limitation is a leadership choice
He explicitly stated, not yet. The four PRs stay held until agent quality and testing significantly improve. So he doesn't trust it. The limitation is a leadership choice. It is a strategic pause. You cannot call the architecture structurally broken when the architect is actively choosing to keep the safety break unengaged while they tune the engine. I think we have fundamentally different definitions of what constitutes a system. A system includes its deployment and its constraints. If the tools exist, but are held back because leadership doesn't trust the agent quality, that proves my point. The testing strategy itself is flawed.
Deep tests against shallow ones
How so? Look at the tension between deep, expensive tests and shallow, broad ones. Prop flow relies heavily on about 51 shallow, cheap checks. But then you have Maestro. Ah, Maestro. And we should be precise here as the report notes. Maestro is Clara's maintenance routing brain. Right. The deep checker being used on it is actually called Morpheus for the runner and Sherlock for the honesty checker. Right. Morpheus and Sherlock. I know they're thorough, but I'm actually a bit confused about how they differ from the basic check mechanically. Are they just pinging a server differently? No, it's a completely different paradigm. Okay.
What a 200 OK does not prove
A shallow check just pings an API. It sends a digital request to a server and if the server responds with a 200 OK status, the shallow check says, great, it worked. It's fast and cheap. Sure. Morpheus and Sherlock use browser automation. They spin up a literal invisible Google Chrome browser, physically log into the prop flow dashboard, navigate to the apholio vendor page and take actual visual screenshots of the work order. Oh, wow. Yeah, then an AI looks at the image to verify the UI is in blank. Which is incredible. But let me ask you this. If the shallow checks are so capable, why did it take these deep, slow screenshot checks to catch the bug where photos were in the database, but silently not rendering on the screen?
Well, or the bug where a shallow check saw that 200 OK status and thought was a success. But the deep check looked at the screenshot and realized it was secretly just a stale login page. Because at folio, the third party software serves its login form at the exact same web address the real page lives at once a session expires. A shallow API mock would never catch that because the server technically responded successfully. I can see at that point. Right. So the shallow tests are structurally blind to real world deployment failures. Sometimes the deep tests are the only things proving the system actually worked. Yet even with all that power, they don't block a merge.
They run after the fact and just post a message to a slot channel. That's a compelling argument. But if you considered the cost of that depth, depth is a luxury you can only afford when your breath is secure. And right now, their breath is compromised. OK. Look at the myestro deep check documentation. The cost is listed as five to ten minutes unmeasured in dollars. You cannot gait a modern CI assembly line on unmeasured 10 minute browser automation for every single minor code change. You'd grind development to a halt. Yeah, that would be slow. Extremely slow. And while you're praising the deep screenshot tests for 11 maintenance scenarios, look at the glaring gaps in their shallow breath.
The mass-text tool nobody grades
You mean the ungraded surfaces again? Exactly. They have a property manager decision tool that allows the AI to send a mass text to an entire building. A mass text. The classifier standing between cancel and confirm on that mass text has absolutely no regression test protecting it. Which is terrifying. It is. A false confirm there would be catastrophic. Well, look at the cadence and re-engagement nudges. The automatic follow-up text clarifies sense to prospects. Right. The deliberately have zero live grading. Because wiring it in would cost a real AI API call on every single automated nudge, which is just too expensive. So they just skip it.
If a database outage happens and a fallback text repeats itself 20 times to a prospective renter, nothing in the real time check or the roll-up review will ever notice. We cannot prioritize unmeasured, slow, deep screenshot tests. When a false confirm on a mass text remains totally unprotected by even the cheapest, shallowest check. But that's exactly why I say it's a structural failure. You have a system where you are bleeding coverage everywhere. Your mass text has no regression test. Your maintenance decisions are ungraded. The check that does cover multiple domains gets skipped 199 times out of 200 to save money. But.
The one deep truth, demoted to a Slack message
And the one place you actually have deep verified truth, the Morpheus and Sherlock screenshot test, it's relegated to an advisory Slack message. The structure is completely inverted. The things that matter aren't checked and the things that are checked don't have the authority to stop the deployment. It is not inverted. It is simply incomplete. You are describing a house that is 90% framed and complaining that the wind blows through the living room. Let's look at what is already working, which proves the structural integrity of the parts. Okay. I mentioned the golden data set in the pinned judge. But look at the bridged test suites.
What the suite gets right
About a third of their test files pull the real live prompt text directly from the production code rather than relying on a developer to hand type a copy. Which prevents the test from drifting away from the reality of the prompt. Exactly. It literally cannot go stale on wording. In the week of this report alone, two live examples showed a production wording change landing in the exact same commit as its matching test update. That proves the discipline works. That is good practice. Furthermore, they built an honesty guard that watches specific folders to ensure a prompt change always ships alongside a test covered change.
And when that broad safety net check actually does run that one in 200 times it gets past the budget cap, it works perfectly. But it's one in 200. Yes, but it correctly caught six real behavior problems on a change that had nothing to do with those areas. The grading logic isn't the weak link. The plumbing is. I hear you arguing that the individual components are elegant. But elegance and isolation is not safety. When I look at all of this, my takeaway is that a system where the broadest check runs half a percent of the time is providing a false sense of security. I just a system where the voice safety nets are deliberately coated with soft flags to ignore failure so they don't break the build is providing a false sense of security.
Gauges not connected to the brakes
They have built an elaborate dashboard of dials and gauges but none of them are connected to the steering wheel or the brakes. I hear that, but for me, the takeaway is entirely different. You have to look at the trajectory. Yes, the historical iterations were chaotic. They threw away verdicts, they had soft flags, they chased false alarms. Very chaotic. But this deep inspection proves that the sixth generation has delivered the goods. They have a 385-case golden data set, they have a calibrated pin judge that doesn't drift, they have bridge test suites that cannot go stale, and most importantly, they have a fully automated end-to-end fixed pipeline.
Waiting on a button press. The fact that a founder is holding those four pull requests back for a strategic review doesn't mean the system is structurally flawed. It means it is highly capable and merely awaiting the flipping of a switch. It is interesting, though, that despite our deep disagreement on the nature of the system, we seem to converge perfectly on the immediate necessary actions. Completely. The report lays out the exact steps and they are indisputable. First, they have to fix the two evil runner bugs. You cannot have the test harness secretly dropping the determinism setting and starving the token budget for newer models like Sonnet 5.
What to do first
You have to measure the AI fairly. Absolutely. Second, they need to turn on the blocking CI gates. Start small. Take the checks that already run reliably, like that it renewals and intent checks. Remove the soft flags and give them the power to block a merge. Let the system actually use its brakes. And finally, they must merge those four pending pull requests. The pipeline from a human thumbs down to an automated injected fix in Clara's prompt is too valuable to leave sitting in a holding pattern. Every day, though, sit unmerged is a day of human feedback just going into the void. We are in total agreement there. The path forward is mechanical, not theoretical.
The question that is left
Which leaves us and our listeners with the real lingering question. When you look at profflo situation, are you seeing the standard messy growing pains of an ambitious AI infrastructure team figuring it out in real time? Or are you looking at a critical failure of safety architecture where the illusion of a safety net is actually more dangerous than having no net at all? There is definitely plenty more nuance to explore in the material, especially when you dig into how these AI models mutate state based on what seem like harmless, plain questions. Indeed, there is. But we'll leave it to the listeners to form their own conclusions.
Just remember, the next time you walk into a building and see that plastic fire alarm on the ceiling, you might want to ask if anyone actually ran the wires. A chilling thought to hand on. Thanks for joining us.