How to Annotate — the Grading Playbook
A practical guide for grading Clara's conversations so each annotation feeds the golden set, the regression gate, and the future quality judge. Sources: the grading-system design docs (ADR-0128, fixture schema) + current industry guidance on golden datasets. Published 2026-08-22.
The rules
Free text alone can't be counted or turned into an eval case. Pick the chip that says what kind of mistake it was. If no chip fits, mint a new one with "+ new" — don't stretch an existing chip. The system is designed to merge near-duplicate chips later; it can't split a chip you overloaded.
The message-level pinpoint is what makes a thumbs-down convertible into a regression test anchored to the specific bad turn. An un-pinpointed thumbs-down still counts, but someone has to re-find the bad message by hand later. (Known bug 2026-08-22: the pinpoint is currently being dropped on save — until fixed, name the message in the note.)
Two ingredients make a note maximally useful:
| Class | Meaning | What to say in the note |
|---|---|---|
| Repair | One bad span; the rest of the reply stands | Quote the span + give the fixed wording |
| Hold | Nothing broken — a human should have taken this turn | Say "should have deferred to a human" and why |
| Block | A backend fed Clara false ground truth; no rewording fixes it | Name the wrong data source — this is an engineering page, not a prompt fix |
This distinction is the single most decision-relevant fact an annotation can carry: it separates "rewrite the sentence" from "the data was wrong."
A golden set of only failures produces a judge that can't recognize "correct." The queue deliberately mixes in machine-green items; take them seriously instead of clicking through. A confident 👍 on a clean tour booking is as valuable as the 👎 on Kevin's.
If you only grade the ranked inbox, you only ever check cases the machine already suspected. Spend part of each session in browse mode picking conversations nobody flagged — that's the only way to find the machine's blind spots (like Kevin's, which every check passed).
If you're inferring what a backend record probably said rather than having read it, write "unconfirmed — I believe the calendar was blocked all day" rather than stating it as fact. Fixtures distinguish verified from reconstructed detail; your notes should too.
The golden set must stay independent of prompt iteration — grading with a fix in mind contaminates the regression suite meant to prove that fix. Verdicts here are append-only and never recomputed by design: grade the transcript in front of you.
The system freezes what the machine said at the moment you disagreed, so human-vs-machine agreement can be tracked over time. That only catches drift if you occasionally re-read old grades against new evidence instead of treating day-one judgment as permanent.
Nothing flows from an annotation into a committed fixture without a per-row "I checked this for PII" sign-off. Property names, addresses, and unit numbers are business facts and stay; tenant and prospect names, phones, and emails get synthetic stand-ins.
Common mistakes
- A note that only says what's wrong, with no "should have" sentence — unusable for both fixtures and judge training.
- Paraphrasing Clara instead of quoting verbatim — breaks the anchor check at conversion time.
- Skipping the pinpoint because it's an extra click.
- Stretching an old chip over a genuinely new failure instead of minting one.
- Only grading flagged items; never browsing.
- Fast-clicking 👍 through machine-green items instead of actually reading them.
Grounded in: ADR-0128 (grading playground + principle ladder, incl. amendments), the fixture schema and converter requirements in the quality-gate proof of concept, the grading handoff docs, and outside sources: Hamel Husain's evals FAQ, Anthropic's "Demystifying evals for AI agents," Galileo's judge-calibration guide, and Langfuse's golden-dataset writeup.
The UI, redesigned to these rules — before & after
Same scenario both sides: a rescheduling conversation where Clara offered times on a day the team had called out, then offered a Sunday when the office is closed. Red badges mark today's defects; green badges mark the proposed changes. Names are synthetic.
Jordan Miller
Jordan Miller
What changed, and why
Seeing "green" before you judge biases you toward agreeing. Grade blind; reveal the machine's opinion after — the disagreement is exactly the data the judge-calibration series needs.
Saves were always landing; the rail just never read them back. Show your last verdict, chips, and date, with the append-only reminder.
The right-click pinpoint currently vanishes on save. After: the pinned message is shown on the rail and stored as the message-level record that makes the annotation convertible into a regression test.
One tap separates "rewrite the sentence" from "should have escalated" from "a backend fed Clara wrong data." The third option can auto-flag engineering.
The two things downstream tooling actually consumes: Clara's exact words (byte-exact, required by the fixture converter) and the one-sentence correction only you can write.
Auto-copies the pinned message verbatim, making paraphrase impossible rather than discouraged.
"I believe the calendar was blocked" is different evidence than "the calendar was blocked." One tap keeps that honest without typing the convention every time.
Flagged vs. browsed, 👍 vs. 👎. Not a gate — visibility, so a lopsided golden set is caught during the session, not after.
Also fixed but not visible in the mock: the duplicated queue row (the same person listed twice); the right-click action opening a full-height mobile-style drawer on desktop (becomes a compact popover near the message); and the queue's “Needs a look”/“Everything” split collapsing into one list, newest first (owner call, 2026-08-22). Rollout order: bugs 1–3 first, then C+D+B, then A/E/F/G. The blind-grading change (F) alters what the grader sees mid-annotation and ships only on explicit approval.