Using our golden data to make Clara better — industry practices, no model training

Research report · 2026-08-18 · requested by Fede

Using a Golden Dataset to Improve Clara Without Fine-Tuning

Research brief — what the industry actually does with labeled example sets to make a production LLM agent better, when fine-tuning is off the table, plus a scoped-in question on how comparable companies structure their knowledge stores. Written against Clara’s real setup: 1,077 labeled emails out of 56k spanning 8 years, the grading playground (ADR-0128), the taught-knowledge lane, and today’s policy storage — one DynamoDB knowledge record per property, organized in sections, staff answers folded in by topic, newest edit wins, no expiry or conflict detection.

The one-sentence framing

There is no single “training on the golden set” move. There are four distinct techniques that all consume a labeled dataset but produce different artifacts: (1) hand-picked examples pasted into the prompt, (2) a live lookup system that finds the most similar past case at request time, (3) written-down rules extracted from patterns across many cases, (4) an automated search process that rewrites the prompt itself and checks its work against the dataset. Clara already does a version of #3 (grading playground) and #4-lite (corpus re-run gate). The gaps are #1 and #2.

1. Few-shot exemplars from a golden set: static vs. dynamic retrieval

What “static” means: a human (or an LLM asked to help) hand-picks 3–5 examples and pastes them into the system prompt, permanently, for every request.

What “dynamic” (kNN/retrieval) means: at request time, embed the incoming email, search the golden set for the nearest neighbors by similarity, and inject only those into the prompt for that specific call. This is the “KATE” pattern from early GPT-3 in-context-learning research — retrieving k nearest neighbors per input beats a fixed example set on most benchmarks.

How many, and how to format them — this part is settled industry consensus, not a single vendor’s opinion: - Anthropic’s own prompt-engineering docs are explicit: use 3–5 examples, wrapped in <example> / <examples> XML tags so the model can tell them apart from instructions. Examples should be relevant (mirror the real use case) and diverse (cover edge cases, vary enough that the model doesn’t lock onto an unintended pattern like always using the same tenant name or dollar amount). - This 3–5 number shows up independently in the broader few-shot literature too — it’s not an Anthropic-only claim.

The nuance that changes the recommendation — and this is where naive “just retrieve the closest match” advice breaks: - Research on similarity-based retrieval found a paradox: on some tasks, the closest matching example is the worst choice, because the model tends to copy the retrieved answer’s surface form even when it’s wrong for the new case. For Clara this is a real risk — retrieving the single most textually-similar past email risks Clara copying that reply’s specific unit number, dollar figure, or promise almost verbatim into an unrelated situation. - There’s also a hard ceiling on “more examples = better.” Studies on many-shot in-context learning found performance plateaus around 50–70 examples and then degrades — irrelevant or redundant examples in context measurably hurt accuracy (one study found ~10% degradation from irrelevant examples), and larger example sets increase output-format errors because the added length distracts the model from the required structure. Curation quality beats example count, consistently. - There’s a second failure mode worth flagging explicitly to a team that just built a “grading playground”: adding examples can silently make behavior worse rather than better, and this is not something you notice by reading the prompt — you only catch it by re-running the eval set. This is exactly why the corpus re-run gate you already built (ADR-0128) exists, and it’s the single most important guardrail in this whole space — most public prompt-engineering advice does not include this gate, and it’s the reason things go wrong for teams that skip it.

Verdict on static vs. dynamic for Clara: static curated exemplars (small, hand-picked, refreshed periodically) are lower-risk and match what Anthropic’s docs recommend out of the box. Dynamic per-request retrieval is a genuine upgrade in principle (matches the right example to the right situation) but adds infrastructure (embeddings, a vector index over 1,077+ growing cases, latency, and — critically — a way to make sure the retrieved example is actually correct, not just similar) and a new failure mode (parroting a near-miss). Recommendation below treats this as experiment #2, after the distillation lane is exploited further.

2. RAG over solved cases (case-based reasoning) vs. distilled rules

Two different retrieval targets exist in the literature, and it’s important not to conflate them: - RAG over raw solved cases: retrieve the 3-5 most similar past labeled emails (with their correct handling) and give the model the raw case. - Distilled rules / principles: extract the pattern across many cases into a written rule once, and give the model the rule (this is what Clara’s grading playground already does).

This is an active academic area under the name “case-based reasoning (CBR) for LLM agents.” A 2025 survey (arXiv 2504.06943) frames the trade-off cleanly:

Retrieval over raw cases (CBR/RAG) Distilled rules (principles/constitution)
Best when The situation space is long-tail / highly varied — many one-off scenarios that don’t reduce to a clean rule (an odd fair-housing edge case, an unusual maintenance-liability dispute) The situation is common and the “right answer” is stable across instances — the same judgment call recurs across many tenants/properties
Explainability Cases are more persuasive/interpretable to a human reviewer than an abstract rule — “here’s the closest thing we did before” is easy for a PM to sanity-check A rule is more compact and auditable as policy, but harder to spot-check against reality without re-deriving it from cases
Maintenance cost Grows automatically as new cases are logged; no synthesis step required, but the corpus can bloat and get noisy without pruning Requires an explicit synthesis/promotion step (this is exactly ADR-0128’s job) — expensive per rule, cheap at inference time
Token cost per request Pays retrieval + injecting full raw case text every time Pays once at authoring time; near-zero marginal cost per request (a rule is a sentence, a case is a full email thread)
Failure mode Copying the near-miss (see §1); stale cases (policy changed, case didn’t) Over-generalizing from too few examples into a rule that’s wrong on the next edge case

Industry read: production support/ticketing systems (the closest commercial analog to Clara) overwhelmingly default to RAG over the raw ticket/case history for exactly the situations that are too varied or too rare to justify writing a rule, while using rules/policy documents for the recurring, stable judgment calls. Neither replaces the other — they’re complementary layers, and several 2025-2026 papers (e.g., an ACM SIGIR paper on customer-service QA, arXiv 2510.08667 on ticket resolution) explicitly combine both: rules/knowledge graph for structure, raw case retrieval for the long tail.

Applied to Clara’s numbers: 1,077 labeled cases is enough to distill a solid principles library (which is what’s happening), but it is not dense enough to make raw-case retrieval reliably find a close match for most incoming emails — 1,077 cases sampled across 8 years and presumably dozens of properties and intents means the average “nearest neighbor” for a random new email may not be all that near. This argues for growing the labeled corpus (or narrowing retrieval to a specific intent/property first) before raw-case RAG pays off, and for distillation remaining the primary lane for now.

3. Principle distillation into the system prompt — the “constitution” pattern

This is the best-evidenced pattern in the whole space, and it’s the one Clara already runs.

Where Clara’s current design sits relative to the field: graded conversation examples → versioned principles → gated promotion into the prompt is ahead of where most teams are — most teams either hand-edit prompts with no eval gate, or fine-tune (which Clara has explicitly ruled out). The main gap is upstream: how examples get selected for grading in the first place (see §7).

4. Automated prompt optimization: DSPy, GEPA, OPRO, promptfoo

These are tools that take a labeled dataset + a metric and use search/optimization to rewrite the prompt automatically, rather than a human writing rules by hand.

Maturity and fit assessment — being honest about evidence quality: - DSPy is the most mature (multi-year, large open-source community, several published production case studies) but was designed for structured pipelines with clear metrics, not necessarily “did this reply handle a nuanced fair-housing-adjacent tenant email well.” It fits best where Clara already has a crisp pass/fail or scored grade — which the grading playground already produces. That’s the actual opening: DSPy/GEPA could optimize against the grading playground’s own scores, using the 1,077-email golden set as the training signal. - GEPA is newer and less battle-tested in the wild (one named production case study found, Decagon’s), but its core idea — reflect on full failure traces in natural language rather than a single number — is structurally very close to what the grading playground already does by hand (a human distills principles from graded examples). GEPA is arguably “the grading playground, automated and run continuously,” which makes it the most philosophically aligned option of the three, but also the least proven at Clara’s evidence bar. - OPRO is the least directly applicable — it’s the original proof-of-concept, superseded in practice by DSPy/GEPA for anything beyond a toy prompt. - None of these tools have a large body of evidence on emotionally-loaded, policy-sensitive, multi-turn property-management email — the published wins are mostly on benchmarks (math, classification, routing) or narrower support tasks. Treat any of them as promising infrastructure to pilot small, not a proven fix.

5. Dynamic prompt assembly: per-property taught knowledge + per-intent exemplars + provenance

This is “context engineering” — assembling the system prompt at request time from parts (instructions, per-property policy, retrieved examples, tool state) rather than one static block.

6. Failure-example usage: do negative exemplars help or backfire?

Mixed evidence, and the failure mode is well-documented enough to be a real caution, not a hypothetical:

Applied to Clara: the grading playground’s actual output format (a versioned principle, not a bare prohibition) is already the safer pattern described above. The risk area to watch is if incident postmortems get fed in as raw “never do X” bullet lists outside that pipeline (e.g., pasted directly into the prompt during a fire) — that’s exactly the shape of input most likely to backfire, and it’s worth an explicit house rule: incident lessons go through the same graded-example → distilled-principle → eval-gated-promotion pipeline as everything else, never a same-day raw prompt edit.

8. Policy-store architecture: one tier or two?

The founder’s question: should curated source-of-truth knowledge (lease terms, the property website, onboarding questionnaires) live in a different store, with different rules, than knowledge that emerges from live operation (a manager’s one-off answer to an edge case, an exception ruling)? And what do comparable companies do?

The two kinds of knowledge are genuinely different, and the difference matters operationally

That scope ambiguity is the actual risk, more than storage location. A one-off exception, once folded into “the policy” undifferentiated from lease-derived rules, can get applied to the next resident who asks — which may be exactly wrong (fair-housing consistency arguments cut both ways: consistent application is required, but so is not manufacturing a policy nobody actually decided).

What the comparable companies do — evidence quality varies a lot, most of this is vendor marketing/help-docs, not independent audits

Cross-cutting pattern (this is the actual finding, not any single vendor)

None of the four appear to run genuinely separate storage tiers with different data models for “static” vs. “emergent” knowledge — that specific split (literally two stores/tables) doesn’t show up as a named pattern anywhere in what’s public. What does show up consistently, across all four and the general enterprise-RAG literature:

  1. One underlying knowledge store, but a human-review gate on anything auto-generated or emergent before it’s treated as settled. Sierra’s staged promotion, Decagon’s “review rather than implement,” Intercom’s contradiction-review tool, and Clara’s own corpus re-run gate for principles are the same shape of control, applied to different content types. This is the strongest, most repeated pattern — call it industry near-consensus, even though sourced mostly from vendor material rather than academic papers.
  2. Provenance metadata, not physical separation, is what the general RAG-governance literature (§5 above) actually recommends for telling static from emergent content apart: tag every fact with where it came from (a lease clause vs. a manager’s one-off answer), when it was added/last verified, and — the part Clara is missing today — an explicit scope marker (this-tenant-only exception vs. this-property-standing-policy). A conflict-detection pass (Intercom’s tool is the cleanest published example) then runs within the single store using that metadata, rather than needing two stores to compare against each other.
  3. Conflict resolution is universally “surface it to a human,” never automated newest-wins, in every system with published conflict handling (Intercom explicitly; the general RAG literature explicitly says silently picking the newer document is a known failure mode — the correct move is “abstain or surface the conflict,” not silently overwrite). This is the one place where Clara’s current “newest edit wins, no conflict detection” design diverges most clearly from what every comparable system does once it matures enough to have the feature at all — though it’s worth being honest that newest-wins-with-no-detection is also probably where most of these companies started, before shipping the review tooling described above.
  4. Expiry/freshness is handled as a metadata field (last-verified date), not a separate short-lived store — recency is a retrieval-time filter/deprioritization signal, not an architectural boundary.

Recommendation for Clara

Given today’s design (one DynamoDB record per property, sections, staff answers folded in by topic, newest-wins, no expiry/conflict detection) and the pattern above, the evidence doesn’t support splitting into two physically separate stores — no comparable company appears to do that, and it would add a real cost (every read now needs to merge two sources, and the ordering/precedence question doesn’t go away, it just moves to read time). The gap that actually matches what mature systems do differently is metadata, not architecture:

  1. Add a provenance field to every section/entry: source type (lease/website/onboarding vs. staff-taught), who supplied it, when, and — this is the one Clara is missing that every published system treats as necessary — scope: does this apply to one resident/case, or is it standing property policy? Taught-knowledge answers should be forced to declare scope at capture time (a one-line prompt to the manager: “is this a one-off for this resident, or should Clara treat this as policy going forward?”), not inferred later.
  2. Add a last-verified/last-touched date per section, even if nothing does anything with it yet — it’s what makes a future freshness pass or staleness flag possible without a migration.
  3. Replace silent newest-wins with a lightweight contradiction check before an update is folded in — even a simple “does this new staff answer conflict with an existing section in this same property record” check (LLM-judged, cheap, off the same infra as the grading playground) surfaced to a human reviewer rather than auto-merged, mirrors Intercom’s contradiction tool and Sierra/Decagon’s “review, don’t auto-implement” pattern, and is the single highest-leverage change given how thin the evidence is for anything more elaborate.
  4. Treat property-level policy promotion the same way ADR-0128 already treats principle promotion: taught knowledge that gets marked “standing policy” (not one-off) should pass through the same kind of gate before it’s trusted the same as lease-derived content — this is architecturally consistent with what’s already built, not a new system.

This keeps the one-record-per-property model (which nothing in the research says is wrong) and closes the actual gap: no scope tagging, no conflict detection, no freshness signal — not “wrong number of tables.”

9. Recommendation for Clara, ranked by expected value

Ranked against what already exists: grading playground (principle distillation, eval-gated), 1,077-email golden set, taught-knowledge lane (per-property policy injection).

Tier 1 — highest expected value, lowest new infrastructure:

  1. Feed the golden set’s coverage gaps back into the grading playground, systematically. Right now the playground distills from whatever conversations get graded; the research above (§2, §3) says distillation is Clara’s best lane, but it only works as well as the examples going into it. Concretely: cluster the 1,077 labeled emails by intent/property/outcome, find which clusters have never produced a promoted principle, and prioritize grading sessions there. This is process, not new code — it multiplies the value of infrastructure you already built rather than building anything new.
  2. Pilot automated optimization (DSPy or GEPA) against the grading playground’s own scoring function, on a narrow slice. Since the playground already produces a graded signal per conversation, that signal is the metric an optimizer needs — no new labeling required. Don’t run this portfolio-wide; pick one narrow, high-volume intent (e.g., a specific maintenance-triage sub-case) where you have enough labeled density in the 1,077 to actually train/validate a held-out split, and let the optimizer propose instruction phrasing changes that must clear the existing corpus re-run gate before promotion, same as any human-authored principle. This tells you, cheaply, whether automated optimization beats a human distilling principles by hand — before investing further.

Tier 2 — real value, needs new infrastructure, sequence after Tier 1:

  1. Small static exemplar sets (3–5, Anthropic’s own number) per major intent, curated by hand from the golden set, refreshed on a schedule (e.g., quarterly, or triggered when a principle gets superseded). Lower-risk than dynamic retrieval, immediately actionable, and it’s the one technique from this whole report that Clara hasn’t touched at all yet despite having the labeled data sitting right there.
  2. Dynamic per-request retrieval (kNN over embeddings) — hold until the corpus is denser. At 1,077 cases across 8 years and many properties/intents, most incoming emails won’t have a genuinely close neighbor, so retrieval quality will be inconsistent and risks the “parrot the near-miss” failure mode from §1. Revisit once either (a) the golden set has grown substantially, or (b) retrieval is scoped to a single well-populated intent rather than searched globally.

How to measure both Tier-1 experiments against the golden set, concretely: - Hold out a fixed slice of the 1,077 (e.g., 20%, stratified by property and intent so rare cases aren’t lost) as a frozen validation set never used for grading-playground promotion or optimizer training — this is the number that must not move backward, exactly the discipline the corpus re-run gate already enforces for principle promotion. - Score both experiments the same way the grading playground already scores conversations, so results are comparable to the existing bar rather than a new bespoke metric. - For experiment 2 (optimizer pilot) specifically: report the eval-set score before and after optimization, plus a manual spot-check by whoever currently reviews graded conversations — automated optimizers are known (§4) to sometimes win on the metric while producing prompts that are less legible to a human reviewer, which matters given the team’s practice of reviewing every conversation by design.

Sources

PropFlow Docs