A space where production data — conversations, renewal decisions, maintenance judgments, funnel runs — flows in continuously and humans grade it 👍/👎. Grades distill into principles (repeatable common sense, not a pile of memorized do/don'ts), and principles drive prompt improvements that are tested against the graded corpus before they ship.
Every decision below is clickable and per-person — your pick is saved under your name. Pick on all eight and I'll implement exactly what you chose.
Most of the plumbing exists: machine grading already runs per conversation in prod (its verdicts are currently thrown away), a judge-based data-pull pipeline exists but has never run, and the conversation thread UI (with the "why did Clara say this" rationale block) is reusable as-is. What does not exist anywhere: a persisted human-verdict store, and any feedback UI.
Think of it like reviewing game film. The plays (real conversations) already get recorded — we just never kept the coach's notes. This builds the notebook, the film room, and a rule that every new playbook change has to be tested against the film before the team runs it.
Two honest shapes. Both reuse the existing conversation thread component and the brand arsenal (glass cards, badges, pill tabs, Phosphor icons).
A is a workbench: sit down, work through the pile, every grade lands with full context on screen. B is a scrolling feed like social media: easy to graze, easy to judge with half the story. Grades are only worth collecting if the grader saw the whole story.
You said "not just conversations — flows, funnels, maintenance, renewals, overall quality." The abstraction that makes that real is a gradeable-item registry (catalog-pattern, like the metric catalog): each item type declares how it renders and what context ships with it. The question is what the thumb attaches to.
Grading the whole conversation is like grading an essay with one letter grade. Grading each message is like margin notes on every sentence. C gives the letter grade always, and asks for one margin note only when the grade is bad — which is exactly when the note is worth writing.
This decides how useful the corpus is for principle extraction. A bare 👎 says "bad" but not why — and "why" is the raw material principles are made of.
If you only collect "bad," a machine later has to re-read everything and guess what you meant. If every "bad" comes with one tap saying which kind of bad, the lesson writes itself. The chips start from a short seed list and grow as you add new ones — the chip list literally becomes the principle list over time.
Camellia alone produces more than anyone will hand-grade. The machine side (the prod grader + the daily-review LLM pass + the dormant judge pipeline) can pre-grade everything and decide what deserves human eyes.
Clara handles hundreds of things a week; a person can carefully judge maybe 30 a day. So a robot reads everything first and puts the suspicious ones on top of your pile — plus a few random normal ones, so we'd catch it if the robot's idea of "normal" drifts.
Your constraint: principle-based, not a giant memorized do/don't list. The distiller compresses graded examples into few durable principles — reconciling contradictions into ranges instead of flip-flopping, bounded in count, versioned. The question is how a principle reaches a live prompt.
A new lesson shouldn't go straight into Clara's mouth. It gets written down, a person reviews it, and it must pass the test built from all your past grades. Once a lesson has proven itself for weeks, you can flip a switch to let it apply automatically — one lesson at a time, never wholesale.
The registry makes "everything that touches Clara" reachable — but v1 has to pick. Shipping two types (not one) is what proves the abstraction isn't secretly conversation-shaped.
Build the shelf so it can hold any kind of box, then ship it with two different-shaped boxes to prove it. Adding box types later is cheap; discovering the shelf only fits one shape is expensive.
/admin/dev/evals/clara/playground), sidenav entry next to ConversationsIf grading is supposed to become a daily habit for the team, it needs a front door — not a room behind the workshop. The dev-tools evals page stays what it is: the test bench.
The existing judge pipeline has a dead suggestedTestCase field — the intent existed, the wiring never did. A confirmed 👎 is exactly the raw material of an eval regression case.
evals/datasets/When you mark something "bad," the system should write up the practice question for you — you just approve it. From then on, every future prompt change gets quizzed on that exact mistake before it ships.
The two new entities, sketched. Final shapes land in an ADR once you've picked above.
// The registry — one entry per gradeable shape (catalog-pattern)
GradeableItemType: 'conversation' | 'renewal_decision' | 'wo_judgment' | 'funnel_run' | …
→ each declares: how it renders, what context ships, which machine pre-grade applies
// The verdict — the thing that does not exist today
GradeVerdict {
itemType, itemRef // e.g. conversation#conv_abc, renewal#saga_123
scope // 'item' | 'message:' | 'step:'
verdict: 'up' | 'down'
principleTags: string[] // required ≥1 on a down (per D3-B)
note?, grader, gradedAt
machineGradeAtTime // what the robot thought, frozen — so we can measure agreement
} | Phase | Ships | Why this order |
|---|---|---|
| 0 | Verdict store + start persisting the machine grades that are currently thrown away | Labeled data starts accruing before any UI exists |
| 1 | Playground UI (layout per D1) with conversations, ranked inbox, verdict rail | The habit starts |
| 2 | Second item type (per D6) + 👎→draft-regression-case (per D8) | Proves the registry; grades start compounding into tests |
| 3 | Principle distiller + versioned principles doc + corpus-gated prompt-PR flow (per D5) | The loop closes — data now improves prompts |
Anything the options above miss — a variant you want mixed, a constraint I don't know about.