How accurate is the conversation topic labeler? Final report

What this is: Every conversation Clara has (calls, texts, emails) gets tagged with a “topic” — leasing, maintenance, tour, renewal, billing, turnover, resident question, vendor, or “no topic” — so the team can filter, search, and see what’s happening on the conversations list and the portfolio dashboard. This report checks how often that tag is right.

How we checked it: We had another AI read each conversation’s messages and judge whether the stored topic tag matches what the conversation is actually about. That’s an automated judge working from message excerpts, not a human reading every conversation — treat the numbers below as a solid estimate, not verified ground truth. We also ran a stress test: re-labeling 25 real conversations a second time to see if the labeler agrees with itself, and throwing deliberately tricky inputs at it (foreign language, spam wrappers, buried requests) to find its breaking points.

1. Headline numbers — and the one fact that changes everything

Across all 2,698 conversations the judge looked at:

Read on its own, a 70% correct rate sounds bad. It’s the wrong number to act on, because it blends together two very different populations that got audited side by side:

• Camellia Apartments — PropFlow’s one live, paying customer, where Clara is actually answering real tenants and prospects.
• Everything else — internal test properties (a property literally named “Propflow.ai Test Property,” a designated test tenant called “The Willows,” and an early pilot property, “Yale 25 Station,” plus its sandbox copy) that the team uses to build and test PropFlow itself, not to serve real customers.

Splitting the same audit by property tells a completely different story:

On the live customer’s real conversations, the labeler is right 94.4% of the time, wrong 2.8% of the time, and the judge was unsure the other 2.8%. Almost all of the “wrong” volume in the blended number comes from test and pilot properties — see why in the next section.

(This property breakdown comes from the underlying per-conversation records we still have on disk, which cover every single one of the 733 wrong cases and about 90% of the full judged set — enough to trust the split completely for the “wrong” bucket, which is what matters here.)

2. What’s actually going wrong — by pattern

Pattern 1 — Never labeled at all (the big one, but not a customer problem)

650 of the 733 “wrong” cases (89%) are conversations where no topic was ever stamped — the field is just empty, not “wrong topic,” just blank. Of those 650:

• 644 are on test/pilot properties, not the live customer. Zero are on Camellia Apartments.
• 561 of the 650 (86%) happened before the labeler even existed. The labeler was built and turned on June 13, 2026; most of these blank conversations are dated April–June 2026, and a big batch is dated May 2026 — before the tool was live. A large chunk are also literally named for testing purposes (IDs containing “gauntlet” or “harness” — 189 of them), i.e., scripted test scenarios seeded straight into the database, which skip the normal message-arrives → get-labeled trigger entirely.
• The team also removed the old nightly “catch anything that slipped through” safety-net job a few weeks ago, on the reasoning that the live trigger was working — so there’s currently no lane that goes back and fills in old, never-touched conversations.

Examples: conv-renewal-tenant-gauntlet-appfolio-45-1779657416937 (scripted test SMS, never triggered labeling), conv_harness_prove_turnover_e2e_2026 (test harness conversation), 002d5104-dbbd-40f1-8d79-88b40ca251ec (Test Property, tour logistics question, never labeled).

Bottom line: this isn’t the labeler picking wrong topics — it’s old test data and pre-launch history that was never run through the labeler at all, and it’s sitting almost entirely outside the live customer’s data.

Pattern 2 — Spam, dead air, and wrong numbers get a real topic instead of “no topic” (8 real-customer cases)

This is the largest genuine mistake pattern on live customer data. The labeler is supposed to say “no topic” for silence, robocalls, wrong numbers, and people who just want a human — but in these 8 cases it instead borrowed a topic from the leftover menu text or picked something arbitrary:

• conv_voice_65da65b6... — pure dead air; “renewal” only shows up because it’s a word in Clara’s own greeting menu, not anything the caller said.
• conv_voice_91355860... — a misdirected Xcel Energy robocall for a completely different company, tagged “vendor.”
• conv_voice_b389112c... — an Amazon delivery robocall, tagged “leasing.”
• Also: a caller asking only to be transferred to a person (tagged “resident question”), a crossed-wire call between two answering bots (tagged three topics at once), a misdirected accounts-payable email (tagged “billing”), and two more ghosted/silent calls tagged “renewal.”

Pattern 3 — Leasing prospects get mistaken for a “unit turnover” (4 real-customer cases)

Four calls where someone is clearly a new prospect asking about apartments got tagged “turnover” (the process of getting a unit ready between tenants) instead of “leasing”: conv_voice_44207cb9... (a straightforward new-lease call), conv_voice_5e686751... (caller literally says “Leasing,” asks about pet policy), conv_voice_85a7a836... (prospect arriving to sign a new lease). This same confusion shows up even more often in the test-property data, so it’s a recurring blind spot, not a one-off.

Pattern 4 — Third-party vendor reps mistaken for leasing prospects (2 real-customer cases)

Two calls from data-aggregator/listing-verification companies (a “Smart Locating” rep and a “Smart Apartment Data” rep, both calling to check occupancy/pricing for their own database) got tagged “leasing” as if they were real prospects: conv_voice_304727d3..., conv_voice_52e620da.... Small in count, but it’s the kind of thing that would quietly inflate a “how many leasing leads did we get” number if anyone trusted it.

Pattern 5 — Topic boundary confusion: billing vs. maintenance/resident question (2 real-customer cases, including a Spanish-language miss)

conv_voice_3d64eb1a... — a call about getting electricity put in the tenant’s name got tagged “billing” (it’s not a payment issue). conv_voice_4785b753... — a Spanish-language call about a maintenance work order (“orden de trabajo”) got tagged “billing” instead of “maintenance.” Worth flagging: our stress test (see Section 4) found Spanish-language input was not a weak spot in synthetic tests — this real example shows the risk isn’t zero in practice, just uncommon.

Pattern 6 — A second topic in the same conversation gets dropped (recurring on test data, worth watching)

A common shape, mostly on test-property data but a plausible risk for real conversations too: a call covers two things — say, a lease renewal and a broken AC — and only one topic gets saved. Examples: conv_voice_1a1759d3... and conv_voice_1aebc884... (renewal discussion + billing question, only “billing” saved), conv_voice_240448dc... (maintenance issue + renewal discussion, only “maintenance” saved). This didn’t show up in the 20 real-customer misses, but it’s common enough elsewhere that it’s likely happening quietly on real calls too.

Pattern 7 — Automated-notification wrappers and buried requests (proven in the stress test, not yet caught live)

Not found in the real-conversation sample above, but proven directly by the stress test in Section 4: when a real request (like a vendor confirming a delivery) is wrapped in boilerplate — the header text a system like AppFolio or Zillow adds to a forwarded message — the actual request can get cut off before the labeler ever reads it, and it guesses wrong. See Section 4 for the mechanism.

3. Where this came from — brief history

One engineer (Gera) built and has owned this almost entirely since it started, June 13, 2026. It’s been through several redesigns as real data revealed problems:

• First version (June 13, 2026): nine topics, and — importantly — the labeler was allowed to skip a message it wasn’t sure about rather than force a guess. The idea was “don’t over-tag,” and it was a deliberate, reasonable call at the time.
• The big rewrite (July 15, 2026): a review of roughly 700 real conversations found about 1 in 5 had no topic — and 95% of those blanks were voice calls that were just silence, dead air, wrong numbers, or spam, not real missed content. The “skip if unsure” rule from version one had quietly become indistinguishable from “the labeler is broken,” because you couldn’t tell a real miss from an honest skip. This is the fix that helps accuracy today: every message now gets a forced answer, and “no topic” became a real, explicit choice with its own reasons attached (dead air, ghosted caller, wrong number, spam, genuinely unrelated) — so the team can finally tell “nothing here” from “the labeler missed it.” The forced-choice change also came with per-topic vocabulary rules built from real examples rather than guesses, and it fixed a class of bugs where a message’s topic accidentally decided which internal record (a tour vs. a work order) a message linked to — that had once caused a prospect’s conversation to wrongly show a “related work order” link.
• A safety-net job got removed (July 16, 2026): the team had a backup job that periodically swept up any conversation the live trigger missed. It was deleted on the reasoning that the live trigger was working reliably. That’s mostly held up — but it also means there’s no lane left to go back and label old conversations that predate the tool, which is exactly what’s driving most of Pattern 1 above.
• A visibility decision worth double-checking (July 29, 2026): the conversations list now hides all five “no topic” reasons by default, including “unrelated” — which is the one reason that specifically means a real person said real things, just not about anything Clara tracks. The team’s own reasoning at the time flagged this risk directly: a genuinely mislabeled real conversation that lands in that bucket becomes invisible on the default view unless someone clicks “show everything.”

4. Stress-test findings — what reliably breaks it

We ran two separate tests against the real labeler (not a simulation):

Test A — does it agree with itself? We took 25 real conversations, re-ran them through the labeler a second time, and compared the fresh answer to what’s stored. 98.1% of individual messages matched. Every disagreement was on a low-content connector line — Clara’s own “let me transfer you” hand-off, a bare “hi, how can I help,” a system log line — never on something a caller actually asked. That’s a strong, reassuring result: the labeler is stable and repeatable on real content.

Test B — deliberately tricky inputs. We took six clean examples (one per topic) and hit each with four kinds of distortion:

Both breaks trace to the same root cause and the same case: the labeler only reads the first roughly 220 characters of each message before sending it to the AI model — a deliberate cost-saving choice, since reading the whole message for every one of thousands of messages would be expensive. Most real messages get to the point fast enough that 220 characters is plenty. But a forwarded AppFolio/Zillow notification email typically starts with 250+ characters of header boilerplate (“New Message Received via Resident Portal,” a reference number, a “do not reply” notice) before the actual content begins — so the labeler never sees the real message at all, and guesses.

Bottom line: the labeler handles language and short/curt exchanges fine. Its one proven weak spot is any message where the real content sits past the first ~220 characters — which is exactly the shape of a forwarded PMS/listing-site notification email.

5. Opportunities, ranked by expected payoff vs. effort

6. What to fix first

Read the whole message before deciding, for email specifically (Fix #1 above). It’s the smallest, cheapest change on the list, it’s the only failure mode we proved with a live, repeatable test rather than inferred from a sample, and email is the message type most exposed to it — Camellia’s mailbox runs through a system that regularly forwards messages with exactly this kind of header boilerplate up front. Everything else on the list is either already small in real-world impact (the boundary-confusion fixes affect single-digit real cases each) or is a product/process decision rather than a labeler fix.

Full per-conversation judge results: verdicts-batch-*.json in this folder. Full stress-test data and logs: adversarial-report.txt, adversarial-results.json, stability-results.json in this folder.

Addendum — benchmark results (same night)

A 500-case stratified benchmark now gates the labeler (PR #5913): 780 gold-labeled Camellia conversations, a judge-labeled sample of the mailbox's full 8-year history (33,725 emails extracted), and 102 adversarial synthetics. Honest numbers after a review round fixed the benchmark itself (channel budget bug, fake majority votes, parse-fail laundering):

Phase 1's email-budget fix delivered +4.4pp overall; mailbox history 69.8% → 84.5%. Remaining named miss patterns queued for the next hardening round: "tour" fired when a showing is merely offered; out-of-office replies inheriting the bounced message's topic; the notification-wrapper pre-rule occasionally stamping a wrapped billing email as leasing; escalation flipping some correctly-quiet threads (no-topic 97.5% → 94.5%). Open framing decision for Fede: whether the 99%/99.9% bar is measured on current live traffic (≈94% today) or the full historical mix (87.8%).

Produced by a 31-agent audit workflow (judge pass over all 2,698 prod conversations + adversarial replay harness against the live labeler). Related: application-link incident (decision D5: significant hardening).