0128 — Grade Clara's production work in one place, and let the graded corpus drive the prompts

• Status: Accepted
• Date: 2026-08-11
• Deciders: Gera (all eight design decisions, answered 2026-08-11 via the decision artifact at https://docs.propflowai.co/a/grading-playground-design.html, propflow-docs commit 0c8700c)

Context

Clara makes thousands of judgements a week — replies on a live thread, renewal decisions, work-order triage, tour/funnel runs — and we throw almost all of the evidence away.

Three separate machines already grade that work in production today, and none of them keeps its verdict:

So the two things we most want are both structurally impossible right now:

1. We cannot tell whether Clara is getting better or worse. Every grade is a fact about one moment that no longer exists. There is no series.
2. We cannot improve a prompt on evidence. Prompt edits are argued from whichever conversation someone happened to read that week. There is no corpus to regress against, so a change that fixes the case in front of us and breaks four others looks identical to a change that fixes everything.

And there is no human grading surface at all. There is no thumbs widget anywhere in the app — not on conversations, not on renewals, not on work orders. When a PM or one of us reads a thread and thinks "that reply was wrong", that judgement has nowhere to go except Slack, where it is not a label on anything.

Why now. The evals we do have are hand-authored fixtures (evals/datasets/*.yaml) written after a bug is found by a person reading a transcript. That loop is bounded by how much anyone reads. Meanwhile production generates the labelled data for free, several times a minute, and we delete it.

The failure mode this design is most afraid of

The obvious version of this feature — "let humans thumbs-down bad replies, and feed the thumbs-downs into the prompt" — produces a prompt that is a growing list of memorised do-nots. That prompt gets longer forever, contradicts itself as cases accumulate, and generalises to nothing. Gera's framing, and the reason D5 is a ladder rather than a pipe: what we want out of the corpus is repeatable common sense — a bounded, versioned set of PRINCIPLES — not a transcript of every mistake Clara ever made.

Decision

Build a Grading Playground: a first-class page where production items Clara touched stream in continuously, humans grade them 👍/👎, and the grades persist as a durable corpus that (a) measures human/machine agreement over time, (b) distils into a bounded versioned principle set, and (c) serves as the regression suite that gates prompt changes.

The eight design decisions below were answered by Gera on 2026-08-11 and are locked. This ADR records them so a future reader can tell whether a piece of code violates one.

The two entities, and why they are two

The goal file sketched a single GradeVerdict carrying both the human judgement and a frozen machineGradeAtTime. Finalising that sketch here: the frozen field only makes sense if the machine grade is its own durable record that the human verdict copies from at verdict time. So there are two entities, and the split is what makes the agreement series measurable:

• MachineGrade — one row per machine judgement, written where the judgement is already made. It unifies three producers that today emit nearly the same shape and persist none of it: the real-time conversation rubric (ConversationGrade: level + reason slug + detail + signals), the daily-review deterministic pre-filter, and the daily-review LLM reviewer (ReviewFinding: severity + reason + detail + attribution + verified citations). One entity, a source discriminator, and citations / attribution optional because only the reviewer produces them. This is the §1 DRY call: three producers of one shape get one store, not three.

• GradeVerdict — one row per human judgement. Carries the verdict, the required principle chips, the optional note, the grader, the scope (D2), and machineGradeAtTime — a frozen copy of the MachineGrade level and source as they stood when the human graded. Frozen, not a pointer: the whole point is to measure human/machine agreement over time, which is impossible if re-running an improved rubric silently rewrites what the machine "had said" when the human disagreed with it.

A third entity, Principle, holds the chip vocabulary (D3) and the graduation state (D5). Chips minted through "+ new" are born status: 'candidate'; the distiller reconciles candidates into the bounded versioned set; promotion to auto-injection is the per-principle human flip.

How a merge works, given that a principle id may never be re-pointed

Recorded here so the code does not keep re-deriving it in a docblock.

Reconciling two chips that name one idea is the distiller's core job. But GradeVerdict.principleTags stores a principle id verbatim in a corpus that is append-only by design, and Principle.id above "must never be re-pointed" — so a merge cannot mean rewriting the verdicts that cite the duplicate. There is no version of that which preserves the evidence.

So a merge is: retire the duplicate, and stamp Principle.supersededBy with the survivor's id. Historical verdicts keep citing exactly what they always cited; every reader that COUNTS principles resolves a tag through the supersededBy chain first and folds the duplicate's evidence onto the survivor at read time. The corpus is never rewritten, and the count is still right.

Two consequences, both load-bearing:

• A reader that skips the fold silently under-counts. The duplicate's citations would be attributed to a retired id that no longer appears in the vocabulary — so a survivor's evidence would appear to drop immediately after a reconciliation whose whole purpose was to consolidate it.
• supersededBy only ever accompanies status: 'retired', and only for the merge case. A principle retired because it was disproven has no survivor and carries no pointer.

Entity classification (ADR-0027)

None of the three is derived. A human's judgement is an original fact — it cannot be recomputed from anything, and if it is lost it is lost. The machine grade is likewise a record of what a specific rubric version said at a specific moment; re-running the rubric produces a new grade, not the old one.

Spine-stamp: the decision-tree call, made out loud

Per docs/architecture/spine-stamp-pattern.md §When to apply, the gate is: does this entity carry human identity — a phone, an email, or a human name?

• MachineGrade carries an itemType + itemRef + propertyId + a machine verdict. Its citations carry message sort keys and quoted text from the cited message. No phone, no email, no name field. It fails the gate.
• GradeVerdict carries the same item pointers plus graderUserId. The grader is staff, and staff identity lives on the User row, which is already spine-stamped. Storing an FK to a stamped entity is not bearing identity — the same reasoning by which WorkOrder is stamped (it carries the tenant's signals) while VendorCompany is not (ADR-0033: the human link lives on VendorMembership, not the company).
• Principle has no human anchor at all.

So: none of the three is spine-stamped, and none appears in spine-stamp-construction-invariant.drift.test.ts. The exemption is by entity type, not opt-out, exactly as the pattern doc requires — and it is recorded here so a future reader does not have to re-derive it.

One caveat that is NOT an exemption, and must be respected in Phase 2: a MachineGrade citation quotes a real tenant message verbatim. That is tenant content living in a new partition. It does not make the row person-anchored, but it does mean the D8 auto-drafted eval case must run the ADR-0097 anonymization transform — the row is a legitimate home for the quote, a public evals/datasets/ fixture is not.

Naming: GradeVerdict the entity vs. GradeVerdict the union

conversation-grade.ts currently exports type GradeVerdict = 'green' | 'yellow' | 'red' — the 🟢/🟡/🔴 level, used in three places in that one file. The entity in this ADR needs that name.

The union is renamed to GradeLevel; the entity takes GradeVerdict. Two reasons, in order: the level is a level (the new name is more accurate than the old one on its own merits), and a repo-wide GradeVerdict that means one thing in the data layer and a different thing in the domain layer is precisely the ambiguity the ONE SOURCE OF TRUTH rule exists to prevent. The rename is 3 call sites in 1 file with no external consumers.

What Phase 0 ships, and what it deliberately does not

Phase 0 is the store, not the surface. Its entire purpose is that labelled data starts accruing before any UI exists — every day Phase 1 takes is a day of grades we would otherwise never have.

Ships in Phase 0:

• This ADR.
• MachineGrade + GradeVerdict + Principle types, interfaces, DDB repositories, JSON store, and registration through the repository seam.
• The real-time conversation grader writes a MachineGrade row per grade.
• The daily-review run persists one MachineGrade row per verified finding (dropped candidates are not grades — they are the verifier working).
• Tests in the same PR.

Explicitly NOT in Phase 0: any page, any route, any sidenav entry, any writer for GradeVerdict (the entity and its store exist; the only thing that writes a human verdict is the Phase 1 rail), the distiller, and the D8 pipeline.

Phasing

Every phase ships as one or more reviewed PRs driven to mergeable. This ADR is the plan document (Lens 25) and is linked from every PR in the series.

Constraints this design inherits

• The item feed reuses the dormant judge pipeline's idea, not its shape. evals/pull-golden-data.ts has the right concept (classify a prod conversation, compare to Clara's behaviour, flag the disagreement) wrapped in the wrong runtime (a launchd script with Telegram and keychain dependencies that has never run here). Phase 1 reworks it into an in-app/Temporal feed; it is not preserved as-is and it is not forked.
• The grading surface reuses the conversation components — ConversationThread (including the ADR-0110 rationale "Why" block), ConversationDetailView, CallAudioPlayer, EvaluationResults. This is also how timestamps land in the property's timezone for free, per the timestamp rule; the desk must not hand-roll a formatter.
• The verdict rail is new UI (there is no thumbs/rating widget in the app) and is composed from the arsenal — GlassCard, Badge, FilterDropdown, TableSearch, BrandTabs, Phosphor icons. PageBody variant="default". No hardcoded colours.
• Multi-tenant by construction — no property or customer identifiers in source; the registry is generic over item type.

Consequences

What this commits us to.

• A new durable write on a hot path. The conversation grader runs on every settled turn on every channel. It now writes a row per grade. That write is best-effort and swallowed on failure, exactly like the existing stampConversationGradedAt — a grader hiccup must never break the turn that triggered it. It is an append-only log with no delete and no TTL, so it grows monotonically; the cross-property read is a bounded recent window on a time-ordered GSI3 sort key, never a full-archive scan.
• The corpus becomes load-bearing. Once D5's ladder is live, a prompt PR is gated on a corpus re-run. A corpus that is wrong — mislabelled, or labelled by one person's taste — becomes a machine for enforcing that error. The 🟢 random sample in D4 is the guard against the ranked inbox teaching us only about the cases the machine already suspected.
• Grades are permanent and attributable. graderUserId means every verdict names a person. That is deliberate (agreement series are per-grader, and a principle minted by one person should be traceable), and it means the rows are staff-attributable records.
• The rename lands in the same PR as the entity, per ONE SOURCE OF TRUTH. There is no window in which both meanings of GradeVerdict exist.

What gets easier. Prompt changes get a regression suite drawn from real production instead of hand-written fixtures. "Is Clara improving?" becomes a query. A 👎 becomes an eval case without anyone writing YAML. Machine rubric changes become measurable — the frozen machineGradeAtTime says whether the new rubric agrees with humans more often than the old one did.

What gets harder. Three new entities to keep registered across the repository seam. A hot-path write to keep cheap. And a standing obligation: every new gradeable item type has to declare a registry entry, or it silently never appears in the queue — the D6 "all types day one" decision is what keeps that obligation from being deferred indefinitely.

Follow-up implied. Phases 1–3 above. Beyond them: retention (the log is append-only by design, and at some point a policy for very old machine grades will be needed — deliberately not decided here), and per-grader agreement reporting, which the data model supports but which nothing in this series builds.

Alternatives considered

Layout (D1). A single-item modal over the existing conversations list was lighter to build but caps the grader at one item per navigation — the whole value is volume, and the desk's keyboard next/prev is what makes an hour of grading produce hundreds of labels instead of dozens. An infinite feed was rejected for the opposite reason: it optimises scrolling over judging, and gives the verdict rail nowhere stable to live.

Unit (D2). Message-level-only grading produces precise labels but is punishing to produce and leaves no answer to "was this conversation good?". Item-level-only is cheap but tells a distiller nothing about what went wrong. The chosen shape asks for the cheap judgement always and the expensive one only when it is informative — a 👎 is exactly the case where the pinpoint is worth the click.

Verdict shape (D3). Free-text-only notes were rejected: they cannot be counted, cannot label an eval case, and produce a corpus only a human can read. A fixed closed chip vocabulary was rejected in the other direction — we do not yet know what the principles are, and a closed list would force every novel failure into the nearest wrong bucket. The "+ new" affordance minting candidates is the middle: the vocabulary grows from evidence, and the distiller — not the chip UI — is what bounds it.

Queue (D4). Gera originally picked the ranked inbox alone, then amended it in-thread to include the browse view. Ranked-only was rejected because it makes the machine's ranking unfalsifiable: an item the ranker scored low is an item no human ever looks at, so a systematic ranking blind spot is invisible forever. Browse-only was rejected because unranked grading spends the scarce resource (human attention) uniformly across items that are mostly fine.

Principles → prompts (D5). Direct auto-injection of every principle was rejected as the memorised-do-nots failure mode described in Context — it is the thing this design exists to avoid. Manual-only (principles as a doc a human consults) was rejected as too weak to change behaviour: it is what we effectively have today. The ladder keeps the human in the loop for every principle's first prompt appearance, and only lets a principle become automatic after it has demonstrably survived the corpus.

v1 scope (D6). Conversations-first-then-see was the tempting scope, and it is what a phased build naturally decays into. It was rejected because the registry pattern is only proved by a second item type — a "registry" with one entry is a hardcoded path with extra indirection, and the second type is where the design's real cost shows up. Sequencing conversations first inside the build is fine; shipping v1 with only conversations is not.

Placement (D7). A dev-tools tab under /admin/dev was rejected: grading is not a debugging activity, the people who should grade are not only engineers, and burying it there guarantees it is used the week it ships and never again.

👎 → regression case (D8). A human-curated queue ("review these before they become eval cases") was rejected as a second grading step layered on the first — the 👎 was the judgement, and asking for it twice halves the throughput. The concern that automation would flood evals/datasets/ with noise is answered by two existing gates rather than a new one: the case arrives as a PR that the normal review bot and CI must pass, and D3's required principle chip means every auto-drafted case already carries a label explaining what it tests.

Amendment 1 (2026-08-11) — retention on MachineGrade.citations: keep them, indefinitely

Status: Accepted · Receipt: f7f2436b6 · Supersedes the deferral in §"What Phase 0 ships" and in the Consequences note that this ADR "covers the D8 export obligation but explicitly defers retention."

The open question was how long MachineGrade.citations may live. Each citation carries verbatim tenant text — a message sort key plus an exact quote — in a new durable, append-only partition with no TTL, and the reflex on any new store of quoted resident speech is to bound it.

Decision: retain indefinitely. No TTL, no field-level expiry, no scrubber.

The decisive fact is one nobody had checked, and it inverts the intuition: every citation quote is a verified substring of a Conversation message that already persists indefinitely in the same production table. Conversation rows carry no TTL. So a citation is not a new disclosure of tenant speech — it is a second, shorter copy of text sitting next door under a key we also keep forever. Expiring the copy while the original remains is privacy theatre: it reduces no exposure, and it buys that nothing at the cost of three real things —

1. It breaks the append-only contract this ADR shipped on. MachineGrade is defined as immutable and never recomputed, because the whole value of the store is the series. A field that silently empties makes an old grade unreadable while still looking like a grade.
2. It destroys the longitudinal purpose. A grade whose evidence has expired cannot be re-judged, cannot anchor a D5 corpus re-run, and cannot support the human/machine agreement series machineGradeAtTime exists to produce. The oldest grades — the most valuable ones for "is Clara getting better?" — would be the first to hollow out.
3. It needs the wrong machinery. DynamoDB TTL deletes items, not attributes, so a field-level expiry means an UpdateItem sweeper cron. This repo names that pattern as the wrong direction in the renewal section of CLAUDE.md ("a cron to compensate for a missing timer is the wrong direction"), and it would be a mutating writer on a store whose defining property is that it has none.

The rider — what actually bounds this

Indefinite retention is correct only while the invariant below holds, and it is the invariant, not a clock, that keeps it correct:

Any conversation erasure or retention path — a tenant deletion request, a future retention policy, an org offboarding sweep — MUST also sweep MachineGrade citations by itemRef.

The argument above rests entirely on the quote's source outliving the copy. The moment a conversation can be erased while its citations survive, that reasoning inverts: the citation stops being a redundant copy and becomes the only remaining record of what a resident said, in a partition nobody thought to look in. itemRef is the join — a MachineGrade anchors on it, so a sweep keyed on it reaches every grade for an erased conversation.

There is no such erasure path in the repo today, which is why this is a rider on a future change rather than work to schedule now. Any PR that adds one is required to satisfy it, and this amendment is the reason to look.

Scope of this decision

It covers MachineGrade.citations only. It does not relax the D8 export obligation: anything that copies a citation out of this store into a shared artifact — notably the auto-drafted eval case — still runs the ADR-0097 anonymization transform first (phones to the +1000 impossible-NPA convention, emails to @example.test, names anonymized). Retention answers "may we keep it"; anonymization answers "may we publish it", and the second answer is unchanged.

Amendment 2 (2026-08-22) — D4's ranked inbox is retired: one list, newest first

Status: Accepted · Owner decision: Fede, 2026-08-22 · Supersedes D4 in full — the row "Ranked inbox by default, AND a grade-everything browse view" in the Decision table above, and every place in this ADR that describes the two-view queue (the ranked "Needs a look" inbox plus the "Everything" browse list, and the seeded random green-sample honesty check that rode the inbox).

D4 picked the ranked inbox, with Gera's own amendment in the same breath ("I still want to see all so I can grade some that maybe didn't get ranked correctly") producing the browse view alongside it. Both shipped. The problem is not that decision — it's what the ranking has actually been fed since: reasonFor (queue.ts) is a placeholder loop-detector plus two narrow compliance checks, not a real judge. Ranking a grader's work by a signal that means nothing yet is worse than presenting no order at all — it teaches a grader to trust a queue position nobody has earned, and it quietly hides that no reliable priority signal exists today.

Decision: retire the ranked inbox, the browse/inbox split, and the green honesty sample. Replace with ONE list — every gradeable item in the window, sorted newest-first.

This also retires the sample's own justification: the honesty check existed because the ranked inbox could hide a systematic blind spot in the machine's confident-fine calls. With every item in one list and nothing excluded, there is no "hidden 🟢" left for a sample to rescue — the whole list is now the honesty check.

The reason classification itself (human_machine_disagree / machine_red / machine_yellow / unrated / machine_green) is not retired — it still labels every row's badge and still drives the reason filter. Only the "and therefore this row outranks that one" step is gone.

What does NOT change

• D1 (the three-pane desk), D2 (the pinpoint), D3 (the required chip), D5–D8 are untouched.
• The reason chip a grader sees on a row means the same thing it always did.
• The moment a real judge exists — the one D4's honesty sample was standing in for — ranking may return, informed by that judge rather than the placeholder rubric. This amendment does not foreclose that; it only removes a rank built on a signal that was never meant to carry this much weight.

Implementation

src/lib/domain/grading/queue.ts, feed.ts, desk-state.ts, src/app/api/grading/queue/route.ts, and src/components/domain/grading/GradingQueuePane.tsx / GradingDesk.tsx — see those files' own headers for the mechanical detail. Each row now also carries its channel (voice / SMS / email) so two rows for the same person on different channels read as distinct conversations rather than an inexplicable duplicate — a legibility fix made necessary by collapsing to one list.

Amendment 3 (2026-08-15, merged 2026-08-23) — Phase 3e: what a promoted principle DOES, and the gate on it

Status: Accepted · Completes D5's fourth rung ("an auto-injected prompt block"). Supersedes the Phase 3d claim (PR #5785, merged 2026-08-23) that "nothing in this repo reads status: 'promoted' to build a prompt today": src/lib/domain/grading/injection.ts does now. That module composes Phase 3d's promotedPrinciples() (src/lib/domain/grading/promotion.ts) rather than re-deriving "what is promoted", so the two answers cannot drift apart.

The gap this closes

Phase 3 shipped the distiller (3a/3b), the corpus re-run (3c) and the per-principle human flip (3d). None of them made a promoted principle change anything Clara says. D5 has four rungs and three were built, so status: 'promoted' was a flag with no reader — which is materially the "manual-only" option this ADR's Alternatives considered section explicitly rejected ("too weak to change behaviour: it is what we effectively have today"). Shipping the ladder without its last rung would have been an unamended reversal of an accepted decision.

The decision: the gate lives on the CONSUMER, not on the flip

A principle is injected into Clara's prompt iff it is promoted and it carries a PrincipleEvalReceipt whose statementHash still matches its current statement.

Both halves are necessary; neither is sufficient. The gate is evaluated at the point of use (src/lib/domain/grading/injection.ts), not at the point of promotion, and that placement is the whole decision:

• A gate on the flip protects one door. A hand-written DynamoDB row, a backfill script, a future admin surface, or a second route written next year all reach status: 'promoted' without passing it.
• A gate on the consumer is the only layer every path crosses. Set the status by hand in the console today and the composed prompt is byte-identical.

The review that shaped this phase named the failure mode it is built against: "if the eval receipt is implemented as a checkbox rather than a verified run id/hash tied to the principle text, the gate is procedural theatre and a button edits Clara's prod prompt." Three properties answer that, and each is pinned by a test rather than intended:

1. PrincipleEvalReceipt has no boolean verdict field. A caller reports case COUNTS; isCleanEvalRun decides. There is nothing to set to true, and a 0 of 0 run is refused rather than passing vacuously.

2. The receipt is bound to the TEXT. The hash is recomputed from the live statement on every injection, so the distiller rewording a promoted principle — its job — drops that principle out of the prompt on the next turn, with no sweeper and nobody remembering to look.

3. A receipt is derived from a run file's contents, not from an argument. src/lib/domain/grading/eval-receipt.ts (pure, tested) refuses on the wrong config, the wrong dataset, a run older than the dataset, different graded wording, rows in neither arm, a scenario missing an arm, a row count that does not account for the fold, an injected arm that did not sweep, and — the one that matters most — an injected arm that did not beat the baseline. A principle that demonstrably changes nothing gets no receipt, so "the evals were green" can never stand in for "this works".

   The honest bound, stated because an earlier draft of this amendment overclaimed it: --run takes a file path. There is no signature, and statementHash is a plain SHA-256 anyone can compute. So this defends against a FORGOTTEN eval, and against the likelier accident of pointing at the wrong run file — not against a fabricated one. Raising that bar means the runner signing its output, which is a separate change with a real key-management story. Describing the current gate as though it were already that would be worse than the gap.

Where the block goes, and why not last

base → principles → capabilities → (denied rail). Composed onto the base prompt and passed through composeWithCapabilityBlock, which strips the denied-applicant rail out of its input and re-appends it at the very end. Two constraints, and no other slot satisfies both: the denied rail must keep the last word (denied-applicant-scope.ts ships a strip-then-re-append purely to hold that position), and a turn-specific override must be able to beat a standing principle.

getUnifiedSystemPrompt's signature is untouched — it is frozen at 16 positional parameters until the options-object migration, and this needed no seventeenth.

Blast radius: a promotion is a fleet-wide act

Principle is a single CONFIG#PRINCIPLES partition and getPrinciples() takes no arguments, so a promoted principle enters the system prompt for every property, every org and every channel that composes it. There is no per-property scoping and none is planned here; the eval bench is appfolio-45 alone, which is a narrower surface than the change reaches.

That is consistent with a principle being a rule about behaviour rather than about a customer (§Spine-stamp), and it is the reason the receipt gate is worth its cost. It is written down because CLAUDE.md's core principle is multi-tenant by construction, and this is the first thing in the D5 ladder that reaches a customer at all. The recorder's dry-run output says so at the moment of the decision, in as many words.

The shipped state, and how it is enforced

Nothing is promoted and nothing is injected in any environment, and that is structural rather than a default someone set: every seed principle is code-defined as active and carries no receipt (pinned), a promoted row without a receipt is refused (pinned), and the block renders '' — so the composed prompt is byte-identical to the one that shipped before this existed. The loader fails soft in the same direction: every error path resolves to '', so a grading-store outage costs a nuance and never a reply.

⚠️ The ladder still cannot move end to end — Phase 3f

Stated plainly rather than left for a reader to discover: nothing in this repository advances corpusRunsSurvived (verified across main and all three open Phase 3 branches). checkCanPromote requires three clean corpus runs, so a first promotion is unreachable regardless of this phase. Phase 3e replaces "a flag with no reader" with a complete consumer; it does not replace the missing counter-bump, which is one link further up.

That bump belongs at the end of a passing corpus re-run (scripts/grading-corpus-run.ts, Phase 3c), must materialise the row it bumps — seed principles are code-defined and are not rows, so a blind patchPrinciple is a silent no-op — and is deliberately not in this change: that file does not exist on main yet, so building it here would mean either a second copy of the corpus runner or stacking on an open PR. It is tracked as Phase 3f and has to land before any first promotion.