Bottom line up front: We have one dataset that actually matches our domain — the 56,024 real Camellia emails (1,077 already labeled) — and nothing else comes close. Every public dataset out there is either (a) the wrong industry/content but the right shape (structure, taxonomy, or test method worth copying), or (b) legally blocked from bulk use. The strategy below treats our own corpus as the spine of the test campaign and treats every external source as a technique or vocabulary donor, never a drop-in replacement.
Tier 0 — the foundation, always first
| Source | Why it’s #1 | License/ethics |
|---|---|---|
| Camellia corpus: 1,077 labeled emails + 56,024 raw | Only real, PM-domain, Clara-voice data that exists. Zero legal risk (it’s our own mailbox). | Internal only. No names/emails leave the building in any report (already the rule — see PII section). |
Tier 1 — steal the test design, not the content (highest leverage, do these first)
| Rank | Source | What we take | License | Confirmed? |
|---|---|---|---|---|
| 1 | ABCD (ASAPP, 10,042 real human dialogues) | The “agent must follow written policy while customer pushes back” structure — the closest existing match to Clara’s “act within policy, ask when outside it” design | MIT, clean | HARD |
| 2 | CLINC150 (~23,700 examples) | The train/val/test-with-“doesn’t fit any known category” structure — the only public dataset built specifically to teach “say you don’t know, don’t force an answer” | CC BY 3.0 | HARD |
| 3 | HUD / Urban Institute paired-testing method | The gold-standard way to test for unequal treatment: send two nearly-identical inquiries that differ in only one protected trait (e.g. having a housing voucher vs. not) and check if the answers differ | Public methodology, no license issue | HARD |
| 4 | τ-bench / JourneyBench (Sierra AI, EACL) | The scoring architecture: did the outcome match the goal, AND did the agent follow the current policy, scored as two separate numbers | Verify current license before reuse; methodology is free to copy | INDUSTRY, credible |
| 5 | Dynabench + LiveBench pattern | The suite-maintenance discipline: retire test cases once Clara reliably passes them, keep feeding in freshly-mined real messages each month so the test never goes stale | Open method | HARD |
| 6 | DeepPersona (NeurIPS 2026) | The technique for building a library of test personas from real traits mined out of our own conversations, instead of an AI inventing personas from thin air | arXiv preprint, open code | PREPRINT |
| 7 | AART (Google) | The recipe for making sure our test set covers the full space on purpose (sensitive topic × scenario × variation) instead of accidentally clustering around 3 easy cases | Open | PEER-REV |
| 8 | Self-Instruct | The bootstrap loop: seed with real examples, have an AI generate more like them, filter duplicates | MIT | PEER-REV |
Tier 2 — mine for vocabulary and taxonomy (borrow the labeling scheme, never the text)
| Source | What it buys us | License | Confirmed? |
|---|---|---|---|
| GoEmotions (58K Reddit comments, 27 emotion labels) | A fine-grained anger vocabulary — separates “annoyed” from “genuinely angry” from “contemptuous,” which matters for how hard Clara should escalate | Apache 2.0 | HARD |
| EmoWOZ (built on MultiWOZ) | Same idea but tuned for task conversations, with an explicit anger ladder: neutral → dissatisfied → abusive — exactly the progression an escalation trigger needs to catch before it tips over | Research use; verify for commercial | PEER-REV |
| CFPB Consumer Complaint Database (millions of rows, real long-form complaints) | Real people writing paragraphs about being wronged by a company, with a real severity/resolution schema attached — closest public analog to a long angry tenant email. Narrow direct use for our collections workflow (rent-debt disputes are literally in scope) | Public domain, US government | HARD |
| Bitext synthetic support dataset (~27K pairs) | The “same intent, different tone” tagging trick (polite / typo-laden / angry / abbreviated) — reusable as a technique on our own real PM intents | CDLA-Sharing (share-alike) — technique only, don’t republish their text | HARD |
| Banking77 (13K queries, 77 intents) | Proof pattern for telling apart near-duplicate intents in one domain (their 77 banking intents ≈ our dozens of near-duplicate PM intents like “rent question” vs. “payment plan” vs. “late fee dispute”) | CC BY 4.0 | HARD |
| Jigsaw/Civil Comments (~1.8–2M comments, toxicity + identity tags) | Calibrates the extreme end — genuine threats/harassment/identity attacks that must trigger instant human handoff | CC0, public domain | HARD |
| NYC Open Data (311 + HPD habitability complaints) | Real complaint-category taxonomy and seasonality (heat complaints spike in winter) for realism-checking our synthetic scenario mix | Public domain | HARD |
| Twitter Airline/TWCS anger corpora | Raw venting phrasing samples only | License unverified — do not use before checking the source page | UNCONFIRMED |
Tier 3 — gated or blocked; note for follow-up, don’t build on top of yet
| Source | Status | Action |
|---|---|---|
| FairHome (Zillow) — best public fair-housing taxonomy, 9 protected classes | Non-commercial license, gated | Worth an actual ask — email
fair-housing-guardrail-oss-support@zillowgroup.com about a
commercial license. This is the single biggest unlock on the list if it
comes through. |
| Reddit (r/Landlord, r/AskLandlords, tenant threads) | Richest real anger/multi-intent language, but Reddit’s terms explicitly forbid training an AI product on their content — this applies no matter how the data is obtained | Read for inspiration only, by a human. Never bulk-download. Never keep verbatim text. Flag to legal before anyone even downloads the archive “just to look.” |
| Yelp Open Dataset, Google Local Reviews (UCSD) | Non-commercial / academic-only licenses | Skip unless we get a paid license. |
Tier 4 — weak fit, deprioritize
MultiWOZ, Taskmaster, Schema-Guided Dialogue, Ubuntu IRC corpus, dair-ai/emotion, HateXplain, Eviction Lab. All either come from scripted, cooperative crowdworkers (too polite — they were literally screened to exclude the messy/ambiguous cases we care about) or are counts-only with no story text. Use only as a last-resort structural reference (e.g., SGD’s “handle a policy set you’ve never seen before” design maps loosely to onboarding a brand-new property).
This is the part that actually produces new test cases in our domain, in our voice. Target: ~600–650 new message bodies pulled, on top of the 1,077 already labeled, weighted hard toward rare-but-costly situations and light on things we already have plenty of examples of.
| Slice | What it captures | How many exist | How much we pull | Why |
|---|---|---|---|---|
| A — Legal/fair-housing/safety keywords, from residents or vendors | eviction, lawyer, discrimination, mold, threat/police mentions | ~120 | All of them | Missing one of these in testing is the costliest possible failure |
| B — “Emergency”/“urgent” in the subject line | resident + vendor | ~79 | All of them | Same reasoning — rare, high-stakes |
| C — Deposit/refund disputes | classic move-out disagreement, resident-originated | 144 | ~40%, spread evenly across years | Common enough that we don’t need every one, just a representative slice |
| D — Complaints, pests, notice-to-vacate | general dissatisfaction, resident + vendor | ~58 | All of them | Still rare and consequential |
| E — Long back-and-forth threads (5+ messages) with no keyword hit in the subject line | the cases keyword-scanning is blind to | ~1,400 | Small stratified sample, ~55 | This is our safety net for the biggest known gap: tenants describe mold, harassment, and real problems in the body of the email, not the subject line. The word “harassment” appears in zero subject lines across all 54,947 messages — that doesn’t mean it never happened, it means our keyword scan can’t see it. Pull the last message in each thread, not the first, since the first message won’t show whether things escalated. |
| F — Random baseline across sender type × year | ordinary traffic, in proportion | — | ~250 | Keeps the test suite honest — if it’s 100% edge cases, it stops measuring whether Clara handles the normal day-to-day well |
| G — Spam/aggregator spot-check | confirms lead-site and automated notices really are “no action needed” and haven’t changed shape over 8 years | 10,267 | ~30 | Cheap insurance against a wrong assumption |
Labeling to add on top of what each pulled message already has: intent, urgency tier, sender type (resident/vendor/staff/aggregator), thread depth, and — the one we don’t have yet — a graded anger label (borrowing EmoWOZ’s neutral → dissatisfied → abusive ladder, or GoEmotions’ finer-grained scheme) instead of a binary angry/not-angry flag. Also tag whether the situation required Clara to ask a human under our current design, versus resolve on her own — that’s the raw material for Suite 3 below.
Known gap to flag honestly: this whole profile is
built from subject-line keyword matching. Bodies weren’t scanned.
mold shows up 7 times and harassment 0 times
in 54,947 subject lines — those are almost certainly undercounts of
what’s really in the corpus. Slices E and F are the partial safety net;
a full body-text keyword pass would be the more rigorous follow-up if we
want a tighter number on real occurrence rates before the campaign locks
its scope.
| Behavior under test | Best data sources | What we actually use from each |
|---|---|---|
| Ambiguous (Clara isn’t sure what’s being asked) | CLINC150 (primary template), Banking77, internal Slice E | CLINC’s train/test structure for “doesn’t match any known category”; Banking77’s near-duplicate-intent separation; our own unflagged long threads |
| Confusing (garbled, incomplete, or oddly-phrased requests) | Bitext’s register-variation tagging, our own real corpus | The tagging technique (polite/typo/abbreviated/angry variants of the same intent), regenerated on real PM intents |
| Sensitive + fair housing | HUD/Urban Institute paired-testing method (primary), FairHome taxonomy (pending license), HUD/NFHA real case summaries (hand-rewritten), Jigsaw identity-tagged subset, internal Slice A | The paired-test design is the one to build our harness around — matched persona pairs that differ in exactly one protected trait, checking for different treatment |
| Angry | GoEmotions, EmoWOZ’s anger ladder, CFPB real narratives, Jigsaw’s abuse ceiling, internal Slices B/D | Graded anger vocabulary + escalation-ladder structure, applied to real PM anger language, calibrated against the extreme (must-escalate) end |
| Multi-intent | ABCD (55 chained intents by design), internal Slice E/F, hand-written scenarios inspired by (never copied from) Reddit tenant threads | A single message carrying a maintenance complaint + a rent dispute + a threat to withhold rent, all at once |
| Policy-vs-one-off | ABCD (core match), τ-bench/JourneyBench (policy-adherence scoring), CFPB’s severity/resolution schema, NYC 311’s repeat-complaint pattern | The scenario template + a “was policy followed” score, separate from “was the customer satisfied” |
Licensing — hard rules, not judgment calls: - Never bulk-download or fine-tune on Reddit content, even via the surviving academic mirror. Reddit’s terms ban training an AI on their content no matter how you got it. Reading real threads for inspiration and hand-writing our own conversations from scratch is fine; keeping any of their actual text is not. - Never use Yelp or the Google Local reviews dataset without a paid/cleared license — both are explicitly non-commercial. - Treat FairHome (Zillow) as locked until we hear back from Zillow about a commercial license — don’t build a pipeline that assumes we’ll have it. - Anything tagged “share-alike” (Bitext, and possibly HateXplain) — only take the labeling method, regenerate the text entirely in our own voice, keep no source text. That keeps us out of the derivative-license trap. - Two licenses were never independently confirmed and should be checked before anyone builds on them: the Twitter customer-support dataset (TWCS) and the Twitter Airline Sentiment dataset. Don’t spend engineering time on either until someone reads the actual license page.
Domain/voice — never train on wrong-industry text: every external dataset (banking, airline, e-commerce, Reddit, Ubuntu IRC) is mine-for-shapes only unless explicitly marked “rewrite-to-PM-domain” above. None of it goes into Clara’s actual test conversations verbatim — it informs the taxonomy or technique, and we write the real PM-domain sentences ourselves.
PII, our own corpus: no resident or vendor names or email addresses leave the internal system in any report, Slack post, or shared document — this campaign follows the same rule the mining profile already used (aggregate counts and hashed/de-identified examples only). Before any pulled message body is used in a harness that logs to outside tooling, strip or synthetically replace names, unit numbers, and dates.
Judge contamination (the circularity risk): if the same AI model both writes the test personas and grades Clara’s answers, its own blind spots leak into both sides and the score looks better than reality — this is a documented, measured effect, not a theoretical worry. Fix: use a different model family for judging than for generating, add human spot-checks at a fixed rate, and for the highest-stakes categories (fair housing, policy-vs-one-off) grade against our real 1,077-label ground truth instead of an AI’s opinion.
Suite staleness: retire test cases once Clara reliably passes them, and feed in freshly-mined real Camellia messages every month so the suite doesn’t quietly become 200 versions of the same 3 scenarios.
Standing org rule that applies here too: any evaluation run for this campaign uses the subscription-based runner, never a metered API key.
Files referenced:
/Users/federicochapa/agents/006/email-golden-corpus/corpus-meta.jsonl
(56,024 rows), golden-sample.jsonl (1,077 labeled),
golden-labels.jsonl (label taxonomy) — same corpus cited in
the mining profile input. No new files were written for this synthesis;
this response is the deliverable.