Superseded as the living doc → the Eval & Testing Roadmap (this page stays as the frozen record, 2026-08-19).

The Eval System — Deep Inspection

How PropFlow checks whether Clara is actually behaving — five rebuilds since March, what holds up from this week's meeting, what "Sonnet 5 was worse" actually measured, where the real coverage gaps are, and six decisions for the founders. Two fleets did the work on 2026-08-19: a 20-agent overnight fleet built the narrative below with every load-bearing claim adversarially re-checked against the repo; a same-day, 21-agent sweep independently inventoried all 54 promptfoo suites and re-verified the Sonnet 5 story a second time from the raw saved outputs, not just the write-ups.

TL;DR

• What's genuinely solid: the idea and the parts — a pinned judge, a 385-case golden dataset now sharing a store with the machine's own grades, a brand-new judge-calibration harness, bridged test suites that read the real production prompt so they can't go stale, and a scoring system (shipped 24–36 hours before this report) that grades every message and every finished conversation, voice included.
• What's structurally broken: nothing but ordinary code checks can block a merge; the one check built to catch AI misbehavior across the board only runs once in ~200 tries; voice's two purpose-built safety nets are both incapable of reporting failure right now; three grading systems have historically thrown away their own verdicts; the highest-stakes decisions (was this maintenance call routed right, was this renewal decision right) have a grading slot and nothing feeding it.
• What to do this week: merge the four already-finished pull requests that turn a thumbs-down into a permanent fix; name the new two-layer scoring system the one official standard; fix the two eval-runner bugs before trusting any future model comparison; flip on blocking CI gates starting with the small, reliable checks; fix voice's two broken safety nets — all five are small, mechanical, already-diagnosed fixes, not new builds.

1. How we got here — five generations in five months

The eval system has never sat still long enough to become one thing. Reading the commit history end to end, it's a story of build, get burned, rebuild — which matches almost word for word how Gera described the pattern in this week's meeting: "we had 5 different iterations, and then the 6th one's a good one, and then... bend everything to that sixth version and delete it."

2. The eval census — what Gera claimed, what holds up

In this week's meeting, Gera made two specific claims worth checking against the code, because they're the reason evals got flagged as needing a hard look.

"We had three different eval classes, and they would contradict with each other." The most likely thing he's describing is his own decision doc from a week earlier (ADR-0128), which lays out, in his own words, three separate systems that grade the same real conversation and historically threw the verdict away afterward: a real-time grader that scores each message as it happens, a separate daily-review process with its own severity/reasoning writeup, and a third system that was designed to compare an AI's classification of a real conversation against what Clara actually did — which, per the doc's own words, "has never run in this checkout." The new unified store (called MachineGrade) is meant to make these comparable for the first time — but checking the actual code shows it only pulls together the first system and both halves of the second; the third one (the one that literally never ran) stays deliberately excluded, its idea repurposed for something else later rather than folded in. So the "three classes that contradict" description is accurate as a historical fact, but the fix in flight only unifies two of the three today, not all three.

Separately, a genuinely different "three classes" idea exists elsewhere in the codebase — not about who grades, but about how expensive and risky the check is (a cheap AI-judged reply check, a free deterministic replay check, and an expensive real-system-writing check). Both are legitimate "three classes" stories; worth clarifying with Gera which one he meant, since the fix only applies to one of them.

"Four of these are true. One of them is fake—or not fake, but not human reviewed. Human review is empty." This checks out exactly, and it's not a bug that's going to fix itself: the eval overview page has a filter for "human-reviewed" checks, with its own icon and tooltip, and today exactly zero of the 82 entries in the catalog are marked as human-graded (25 are simple pass/fail checks, 57 are AI-judged). The filter option exists; nothing has ever populated it, and nothing currently can.

A real, still-live contradiction, confirmed today: one specific check (covering how Clara scopes a renewal) is judged a pass at 60% correct by the nightly checker and a pass only at 95% correct by its own dedicated per-change checker — same code, same day, same measured score of 80%, and it's a hard pass on one and a hard fail on the other. The code's own comment calls this "the WORST conflict in the repo — 35 points." A similar, smaller version of this same shape (a change-checker defaulting unlisted domains to a strict 100%-or-nothing bar, while the nightly checker uses a real measured floor) shows up cleanly on at least two other checks (one gated at 90%, one at 89%) and, less cleanly, on roughly 21 others — some of the total 28 flagged cases actually run in the opposite direction, or involve a different pairing of checkers than the simple two-runner story.

A caution worth carrying into every other "these disagree" claim in this system: a dashboard panel used to flag a "disagreement" any time a made-up 100%-or-0% pass/fail number differed from a real measured percentage — which fired constantly, on nothing real. It's since been fixed to distinguish "two real numbers that disagree" from "one real number and one made-up placeholder that were never comparable in the first place." A second, independent case (a credential-swap test) initially looked like a 6.7% quality gap between two different sign-in methods and turned out to be one ambiguous test case getting counted six times over in a duplicated dataset. Two separate false alarms, same root cause: comparing numbers that looked alike but weren't measuring the same thing. Worth applying that same skepticism before treating any other "contradiction" in this system as a real behavioral gap rather than a measurement artifact.

3. The Sonnet 5 story — what was actually measured

There are five separate real-world comparisons involving Sonnet 5 in this codebase, and only one of them is the source of the belief that "Sonnet 5 was worse." It's worth being precise about what that one comparison actually was, because the belief has outlived the nuance.

The precedent that should have been a warning sign. In late July, switching Clara's main model to Opus 5 triggered a same-night comparison that looked bad — Opus 5 lost 8 of 21 maintenance scenarios. That result got formally challenged and refuted: both sides of the test had accidentally run on an old code checkout that was missing two fixes — one for a bug where the code read the model's internal reasoning as if it were the actual answer, and one where the reply budget was too small for a model that "thinks" before answering, so the model silently ran out of room and returned nothing. Once both fixes were in place, Opus 5 climbed from 12 correct out of 21 to a clean 21 out of 21, tying the incumbent. Two real, dated bugs — not a capability gap.

The comparison everyone remembers. In early August, an engineer ran a large, thorough comparison — 48 test suites, roughly 995 individual scenarios — pitting Sonnet 4.6 against Sonnet 5 on Clara's main conversation loop. It lived entirely in an uncommitted local folder: never reviewed, never merged, never seen by anyone but its author. Crucially, it never tested the model actually answering calls at the time (Opus 5 was live the whole comparison window) — it compared two candidates, neither of which was the real incumbent. The raw scoreboard: 35 suites tied, 10 favored Sonnet 4.6, 3 favored Sonnet 5.

The same two July bugs, unfixed, baked into every single result. The eval runner used for this comparison silently drops the setting that makes replies deterministic whenever the model is Sonnet 5 (it's rejected by the API on that model line), so Sonnet 4.6 answered the same way every time while Sonnet 5 sampled differently each run — not a fair fight. Separately, the reply-budget bug from July reappears on at least 9 of the suites: several of Sonnet 5's "losses" turned out to be the model running out of room mid-thought, recovering cleanly once given more space. This exact bug class — the one that already fooled the team once — was still live and unfixed a month later, and remains live in the shared eval code today.

Checked and ruled out: the judge doing the grading was Sonnet 4.6 itself, raising an obvious worry that it might favor its own family. That was specifically tested and the opposite was found — on the one case where Sonnet 5 clearly won, the Sonnet-4.6 judge failed its own model's output. Self-preference bias, if anything, would have pushed the score the other way.

The honest headline, once you read past the summary: the comparison's own follow-up write-ups (not just its top-line report) confirm real, separately-reproduced problems — and there are more of them than the original "3 confirmed regressions" count suggested, not fewer. At least 6–7 distinct issues survive close reading: two separate cases of Sonnet 5 changing real account state (charges, renewal terms) on a turn where the human only asked a question; a one-time but real leak of a tenant's military-deployment details into an internal note; a broken call transfer that sent a caller nowhere instead of to a person; an internal-handoff description leaking into a live reply, which the reviewer's own notes call "not an artifact — the exact production failure this suite exists to catch"; a leak of internal accounting details to a vendor; and a recurring pattern of Sonnet 5 taking an action with zero words said out loud, which shows up across six different suites and holds up under adversarial re-testing.

What that means in plain terms: "Sonnet 5 was worse" doesn't fully survive scrutiny — a meaningful chunk of the headline number is measurement noise from bugs the team already knew about. But it also doesn't fully collapse into "it was fine all along" — a real, separately-confirmed core of behavior problems is there too, and it's a bigger list than the comparison's own summary reported. The honest verdict is neither of the two easy stories: it's a flawed test that still found some real things.

What this comparison never checked, and what's happened since. It never tested the model actually live in production; it skipped the historically most-discriminating test suite entirely; its most in-depth safety/compliance review crashed after only 2 of 19 scenarios, so no safety scenario in that lane was ever scored; and the conversation code it measured has had 50 further changes since — so even its clean findings describe a version of Clara that no longer exists.

Two other, unrelated Sonnet-5 decisions, both fine. Fede chose Sonnet 5 for the internal owner-chat assistant on August 1st, deliberately overriding a benchmark gate it narrowly missed — quality tied, replies came back 24% faster — and it's been live since. Sonnet 5 was also adopted for the turnover-condition extraction step on July 30th after a tie on quality with faster, cheaper results. Neither of these was touched by any of the bugs above; both remain live today with no reported problems.

4. The coverage map — where the blank spots actually are

The two blank spots worth worrying about most: maintenance-decision quality and renewal-decision quality are both declared as gradeable in the system's own data model, and both have zero live inputs feeding them — not disabled, not broken, just never wired up. Every conversation around those decisions gets graded (was the tone right, did the conversation resolve); the decision itself — was this the right call — does not.

5. Voice — two safety nets, neither one can catch you

Voice has more real content-checking machinery than almost any other surface — and, until yesterday, less of it actually reaches anything that can stop a bad call from happening.

The nightly check that's switched off. A workflow purpose-built to actually be able to fail (it phones Clara as every kind of caller, across every stage of a caller relationship) was created just over a week before this report — and switched off at the platform level 32 seconds after it was turned on. It has run zero times since. Nobody documented why.

The nightly check that runs every day and can never fail. A separate, broader voice check runs every single night and has for weeks — but it's built with a permanent "soft" flag, meaning the part of the code that would actually fail the check is never reachable, no matter what it finds. Its own header comment says, in plain terms, don't ever make this a required check while it's soft. One real, complete run of this check found 14 genuinely serious problems — Clara confirming a tour booking after the booking itself had actually failed, booking a tour without the caller ever agreeing to it, offering a time slot on a day the property is closed, and ignoring an explicit rule not to discuss a specific tenant — and the workflow still reported "success."

A paid vendor tool, bought and abandoned. ElevenLabs (the voice platform) has its own built-in scoring feature. It was wired up once, pointed at 3 test scenarios, and hasn't been touched since — over three months. The 21 scoring rules that do exist in the codebase are pointed at an agent that's been retired; none of the 8 agents actually answering real calls today receive them. Whatever scores do come back from a real call get saved to the database and then read by nothing — no dashboard, no alert, only the feature's own internal test.

What's genuinely new and good. The scoring system that shipped the day before this report reaches voice, too — every finished call with a real property now gets the same posthoc scoring text messages get, plus a voice-specific check that compares "did we actually close the loop" against what the call itself recorded. It deliberately skips the fair-housing wording check for voice for now, because judging exact phrasing against a summary (rather than a word-for-word transcript) was judged too unreliable — a real, honest limitation, not a hidden one. This is one day old and hasn't yet been checked against a full day of real call volume.

The net effect: voice has meaningfully more signal today than it did a week ago. But the two checks specifically built to be able to say "this is broken, stop" are, right now, both structurally incapable of saying that.

6. Maestro and the depth-vs-breadth question

A naming note worth getting right before talking to Gera about this. "Maestro" in the codebase is Clara's maintenance-routing brain — the thing being tested. The screenshot-verified checker Gera demoed this week ("it actually confirms it, because now it requires screenshots") is a different system entirely, named Morpheus (the runner) and Sherlock (the honesty checker). Using the names interchangeably will cause real confusion in any follow-up conversation.

What it actually does. At each step of a scripted maintenance scenario, the checker opens a real, logged-in browser and takes a real screenshot of both PropFlow's own page and AppFolio's page for the same work order — not a mock, not a stand-in. It's specifically built to catch fakes: it checks the screenshot isn't secretly a login page (AppFolio serves its login form at the exact same web address the real page lives at once a session goes stale — a check added after the team got burned by exactly that), confirms it landed on the actual page requested, and confirms the page isn't blank. Two real historical bugs are on record that only this depth of check would have caught: photo uploads that existed correctly in the database but had silently stopped rendering on either real screen, and a stale-login screenshot that would have quietly passed as real proof of nothing.

How much of the system this actually covers. Roughly 11 scripted scenarios end to end — a handful hand-written, several auto-generated with a human giving a thumbs-up in Slack before they're trusted — covering maintenance and turnover text flows only. Voice, leasing, and renewals can't run through this pipeline at all.

Cost: genuinely never measured. The team's own design notes list the run time as "~5–10 minutes, UNMEASURED" as of their last update, and nothing in the codebase records an actual dollar figure per run.

Where it sits today: advisory, after the fact. It doesn't stop anything from shipping — it runs after a change has already merged and posts a result to Slack. Turning it into something that blocks a merge is a named, still-open decision, not yet made.

Gera's own question this week, unanswered anywhere in the codebase: "should we have a super in-depth, but low count, maybe only like 5 maintenance evals, or have 30, but it's very surface level? Or a mixture?" There's no written answer to this anywhere — but there is a real number to anchor the conversation with: roughly 51 cheap, fast checks already run on every relevant change or every night, alongside the roughly 11 deep, screenshot-verified scenarios. So today's actual split is closer to "11 deep, 51 shallow" than a literal 5-vs-30, and there are two real, named incidents on record that only the deep tier would have caught — with no incident on record where the shallow tier caught something depth would have missed. That's not proof depth is worth more per dollar — nobody's measured the dollar side — but it's the honest state of the evidence.

7. What's already working

It's easy for a report like this to read as a list of everything broken. A fair amount of what exists is genuinely solid, and some of it is brand new.

The calibration harness. Built the day before this report specifically to answer the question underneath most of Gera's worry: do the AI judges doing all this grading actually agree with a human? It pulls the literal production judge code (not a copy of it) and checks it against a human-labeled set of examples and 262 real conversation turns, with real honesty rules built in — it refuses to count a failed check as anything but a failure, and refuses to print a percentage from fewer than 10 examples. First results: the fair-housing judge got all 85 clean examples right; the general "did this cross a line" judge had zero timeouts across every real turn it checked. Neither number is a finished false-positive rate yet — the harness says so itself — but the instrument to get one now exists and is honest about its own limits.

The pinned judge. After getting burned once in July — a same-night comparison where the judge doing the grading and the model being graded were quietly the same setting, so the test changed both at once — the team pinned the eval judge to a fixed model name, with its own automated check that would turn red if anyone tried to quietly change it. That's exactly the discipline that would have prevented the July incident, and it's held since.

The golden dataset. 385 of Fede's own manually mined judgments — pulled out of old commits, meeting transcripts, past pull requests, chat sessions, and Trello — have been imported into the same store the machine's own grades live in, with 160 of them already tied to a real production conversation. For the first time, a human's opinion and a machine's opinion about the same real conversation sit next to each other in one place, ready to be compared.

The honesty CI guard. A separate, simpler check makes sure a change to Clara's prompt always ships alongside a matching change to its test coverage. It's imperfect — it only watches two specific folders, so it missed a real case where a prompt lived somewhere else — but the idea and the enforcement both genuinely work where they're pointed.

The bridged suites. About a third of the test files pull the real, live prompt text directly from production code rather than a hand-typed copy, so those specific tests literally cannot go stale on wording. Two live examples from this week alone show a real production wording change landing in the same commit as its matching test update — proof the discipline works, not just a design intention.

It works when it runs. The one time out of roughly 200 recent attempts that the broadest safety-net check actually executed, it correctly caught six real behavior problems on a change that had nothing to do with those areas. The grading logic isn't the weak link — whether it gets a chance to run is.

8. The target — one standard eval procedure

Every piece the team keeps describing as the goal already exists in this codebase. What's missing is connecting them into one path, not building anything new.

1. The judge always sees the whole story, never just the reply. This is already true of the newest scoring system — it looks at the full conversation, the tools that were called, and the reasoning behind them, not just what Clara said out loud. It needs to become the rule everywhere, including the older test files that still hand-type a guess at the prompt instead of pulling the real one.

2. A thumbs-down becomes a permanent fix, automatically. The full path — turn a verdict into a written rule, replay that rule against real history to prove it holds up, promote it once it survives three clean replays, and inject it into Clara's actual prompt — is fully built. It's split across four separate, finished pull requests, all green, all reviewed, all sitting untouched for four days because they were deliberately marked "a human has to click merge" and nobody has yet. This is the single highest-leverage thing to do this week: merging four already-finished pieces of work turns "someone left a thumbs-down" into "the fix ships and Clara never makes that mistake again," instead of the verdict evaporating the way three earlier grading systems' verdicts already have.

3. Every pull request shows its own quality delta. Nothing in the repo does this today — there's no comment on a change showing whether it made things measurably better or worse compared to the main branch. This is the missing link between "we have a lot of eval infrastructure" and "an engineer actually sees the number move when they make a change," which is precisely the thing Fede said he wants: "if I make a change, and I push it to CI, it should tell me like, hey, you regressed."

4. Nightly runs, honestly reported. The comprehensive nightly check is already strong when it comes to coverage. Two fixes make it trustworthy: make its per-change sibling actually run more than once in 200 tries, and give voice a nightly check that's allowed to say "this failed" — both are small, mechanical fixes to existing files, not new systems.

5. A model-swap protocol that can't be fooled by its own tooling. Any future comparison between models needs four things, all cheap: fix the two known measurement bugs first (the setting that silently disappears for newer models, and the reply budget that's too small for a model that reasons before answering); compare against what's actually answering calls today, not two candidates neither of which is live; run every case more than once, since a single run of anything is close to meaningless; and pin the judge model for the whole comparison so it can't quietly change mid-test. Every one of these is a fix to something that already exists, not a new build.

The honest summary: this team does not have an evals problem in the sense of missing capability. It has a wiring problem — genuinely good, mostly brand-new parts that don't yet talk to each other or to anything that can stop a bad change from shipping. Fixing that is a matter of days, not months, and most of the work is merging what's already done.

9. Gaps, ranked

Highest-leverage first, per the overnight fleet's own ranking.

1. Nothing actually stops a bad AI-behavior change from merging today — only ordinary code checks (build, types, unit tests) can block a merge; every AI-quality check is advisory, and the broadest one only really runs about once every 200 attempts.
2. Voice's two dedicated safety nets can't currently report a real failure — one is switched off, the other is built so it can never say anything but "success" no matter what it finds; a real run found 14 genuine safety problems and still passed.
3. Any future comparison between AI models on this codebase inherits two live, unfixed measurement bugs — a determinism setting that silently disappears for newer models, and a reply budget too small for models that reason before answering — the same bugs that already produced one false alarm in July.
4. The system that would turn a human's thumbs-down into a permanent, automatic fix is fully built and sitting untouched — four finished, green, reviewed pieces of work waiting on someone to click merge.
5. The two highest-stakes decisions in the whole system — was this maintenance call routed/priced right, was this renewal decision right — have a slot reserved for grading and nothing feeding it.
6. A handful of test suites run in zero automated places at all, and roughly eight more domains have no nightly safety net — only a once-a-day check that only fires for whichever change happens to trigger it first.
7. A real production fix to how Clara flags renewal counter-offers shipped without its matching test being updated, so the test can only check that some escalation happened, not the right one.
8. A bug that could let a stale, meaningless green check quietly hide a real red result on the same code was found and fixed — but shows how fragile the plumbing around "is this actually passing" still is.
9. 35 separate test files each hardcode their own copy of which model does the grading, with nothing checking they match the officially pinned judge — changing the judge on purpose means remembering to hand-edit dozens of files.
10. One specific check is judged a pass at a 60% bar by one system and a pass only at a 95% bar by another, on the same code, the same day — a 35-point gap the code's own comments call the worst conflict in the repo.

10. Opportunities, ranked

Same ranking logic, read as a to-do list instead of a risk list.

1. Merge the four finished, green, reviewed pull requests that complete the path from a human's thumbs-down to a permanent fix — the single highest-leverage thing available this week, and it's already done.
2. Turn on the ability for evals to block a merge, starting with the small checks that already run reliably every time, then expanding — the design is already approved, only the wiring is missing.
3. Fix the two eval-runner bugs (a setting that silently disappears, a reply budget too small for newer models) once, centrally, before any future model comparison — removes the single biggest source of a misleading result.
4. Name the scoring system that shipped this week (grade every message, then roll up every finished conversation) as the one official procedure — it already matches what both founders described wanting, it just needs to be pointed at everywhere and called the standard.
5. Fix voice's two broken safety nets — both are small, mechanical, already-diagnosed fixes, not new work.
6. Wire up the two empty judge slots (was the maintenance call right, was the renewal decision right) — the plumbing for one of them is already sitting in another held pull request.
7. Clean up the handful of test suites that run nowhere — either wire them in or delete them, cheap either way.
8. Add the missing field to the renewal test schema so counter-offer tests can check the actual decision, not just that something happened — a small, targeted fix closing a real, incident-driven blind spot.

11. Decisions for the founders

Answers save for everyone who opens this page; the recommended option is listed first and marked.

DECIDED 2026-08-20 (Fede): not yet — the four PRs stay held until agentic quality and testing significantly improve. Recorded from chat; the widget below remains for the record.

Receipts for Decision 3: the four held pull requests are #5782 (distiller + versioned principles doc), #5784 (corpus re-run gate), #5785 (per-principle promotion arm), and #5786 (the promoted principle reaches Clara's prompt, with an eval receipt attached) — all four opened 2026-08-15 under ADR-0128 D5, all currently green and mergeable (GitHub reports mergeState: CLEAN), and each one carries an explicit "Do not merge — needs a human" marker asking for exactly this decision.

12. Appendix — the full suite inventory

The same-day, 21-agent sweep independently inventoried every promptfoo suite in the repo: 54 suites total, roughly 1,127 test cases, all under evals/promptfooconfig-*.yaml. Two lanes run them — the PR-triggered umbrella sweep (capped at one real execution per UTC day; see the TL;DR) and the comprehensive nightly sweep (not budget-capped). Four suites (knowledge-grounding, leasing-tour-followthrough, tour-confirmation-virtual, triage-vendor-po) are invoked by neither — a human typing the command by hand is the only way they ever run. Eight more (collections-legal, leasing-on-behalf, leasing-specials, message-rationale, unverified-maintenance, vendor-calling, vendor-po-handling, voice-feel) are missing from the nightly sweep specifically, so the once-a-day umbrella gate is their only automated backstop. Two (pm-english, prospect-summary) hand-copy a production prompt constant that has since changed underneath them — confirmed stale, not merely old.

The Sonnet 5 story, re-verified a second, independent way

The same-day sweep re-ran the Sonnet 5 fact-check from scratch — reading the raw saved model outputs behind the disputed comparison, not just its write-ups — before this table was assembled. It confirms the account in §3 above and sharpens two of its numbers.

Method & caveats

Overnight fleet (2026-08-19): 20 agents, ~1.9M tokens, 639 tool calls across nine investigation lanes (history timeline, contradicting eval classes, coverage matrix, dev-tools integration, Maestro, voice evals, and three research lanes), synthesized into the sections, gaps, opportunities, and decisions above. Ten of its most load-bearing claims were independently adversarially re-verified against the repo (not the write-ups): five held exactly as stated, five held on the core fact with a specific, named correction folded into the text above. Same-day sweep: 21 agents across seven lanes (promptfoo inventory, model A/B history, judge architecture, CI wiring, staleness audit, harness inventory, external research), producing the full 54-suite table and a second, independent pass at the Sonnet 5 story — 14 of its claims were adversarially re-checked, all against raw saved outputs rather than summaries; 4 held exactly as stated, 10 held on the core fact with a named correction (both folded into the tables above). Where the two fleets' independent counts of "real, adversarially-confirmed Sonnet 5 regressions" differ from the original comparison's own headline, this page reports the corrected, higher count (6–7), not the original "3 confirmed" figure — the original undercounted. No customer name, phone number, or email appears anywhere in this page; every affected person is described by role. Related pages: the architecture source of truth (the tracker and the five already-decided D1–D5 decisions this page's context box summarizes) · the honesty-layer deep inspection (the sibling investigation this page's structure follows — a different guard system, same investigative method).