Quality Desk Check-In

Is the Cerberus / grading-desk initiative hooked up, is it catching anything, and what is in the way. Plus a UX fleet's findings on simplifying the desk.

2026-08-26 · data pulled live from production · UX review: 6 lenses, 39 findings adversarially verified, 31 survived · Trello: Grading desk stops fighting the grader

Scorecard

577

machine grades in the last 7 days (5 judges, latest ~1 h ago)

red flags on file — real Clara failures

human grades by a real person (all Fede). 159 are the golden import, 2 smoke.

16 / 187

human grades usable by the test pipeline

permanent regression tests landed from 👎

golden-set tests stuck on a merge permission

Is it hooked up and live?

Piece	State	Detail
Machine judges (rubric, topic sweep, policy gate, fair-housing, trajectory, daily reviewer)	LIVE	666 grades on file; ~80/day. Running today.
Human grading desk	LIVE	One real user. Team has not graded a single item.
👎 / 👍 → regression test (nightly)	LIVE · ARMED	Armed 2026-08-23. Runs every morning, clean. 2 tests landed.
Auto-merge of drafted tests	BLOCKED	The merge robot can't see that checks passed (token lacks `Checks:Read`). 3 green PRs unmerged since 08-24. 30-second fix, Fede's side.
Cerberus in front of live outgoing messages	NOT WIRED	It only runs as a test suite in CI. It has never checked a real message before Clara sent it.
Golden-set raw judgments	HALF	59/152 matched to real conversations and shown on the desk. The reasoning text itself is read by nothing.
Promoted principle → Clara's prompt (with eval receipt)	SHIPPED	Landed 2026-08-26.
Overall quality roll-up	MISSING	No view adds the grades up. The desk is one-at-a-time only.

What it has caught

The 85 red machine grades, grouped. These are the failure classes the product vision worries about most: unbacked promises and dropped loops.

Pattern	Count	Real example from production
Clara claims she did something the tools never did	24	"I've got the work order in" — no work order was created. On a call, promised to text a tenant the handyman's timing — no text was sent.
Escalation promised, never handled	14	Promised the leasing team would follow up on lease dates, move-in special, parking, buzz code — none came back to the resident.
Lead dead-ended	11	Thread trails off after the resident's last question; no resolution.
Wrong topic label (bookkeeping)	17	Nightly sweep relabeled resident_inquiry → leasing.
Invented fact	6	—
One-offs	1 each	Missing emergency safety instructions; "closed Sunday" repeated three times; prospect had to ask "what time is my tour?" twice.

Human 👎s: 178 of 195 verdicts are thumbs-down; only 12 pin a specific message. Why the promise failures keep happening — and the one design that fixes the class — is in Clara's promises: the systematic pattern.

Going well · Gaps

Going well

Judges are real and prolific, and they flag the right things.
The learning loop exists end to end: 👎 → test in CI, and promoted principle → prompt, both shipped this week.
The checker's own accuracy is protected by a CI gate that survived a 53-trial burn-in (report).

Gaps

No roll-up. Nobody can answer "how good is Clara this week."
Human grades mostly unusable (171/187): no pinned message or no principle tag — the desk UI makes both feel optional.
Humans grade ~5/day; judges ~80/day. The desk is the bottleneck of the loop, and only one person uses it.
Cerberus is not guarding live sends.
Golden-set reasoning read by nothing.
Merge permission blocking 3 tests.
Burn-in found 4 HIGH ways one person could fake a green CI result — unfixed.

KPIs the desk should show

All computable from data already stored.

Clara quality: green / yellow / red rate per day, property, channel (text, email, voice), judge.
Trend: red rate this week vs last; top principle tags on 👎.
Judge trust: human–machine agreement rate — the rubric's own accuracy score.
Coverage: % of conversations with any judge; % with a human grade; grades per person.
Learning loop: grades usable by the pipeline (16/187 today), tests landed, principles promoted.
Outcome tie: red rate on conversations that ended in a tour/lease vs. went quiet.

UX fleet: how to simplify the desk

Six reviewers (hierarchy, flow, attribution, layout, consistency, copy), each finding then attacked by a skeptic against the actual code. 31 of 39 survived. Ranked by impact.

Change	Effort	Impact	What changes
Show who graded (initials), not a green check	M	high	Initials next to the row: FC / teammate / a distinct "golden import" mark. Grader identity is already stored — pure UI.
Right-click = done	M	high	👍 records instantly. 👎 shows principle chips inline in the popover; tapping a chip submits. "Record verdict" only for the long form.
Kill the mobile sheet on desktop	M	high	Right-click 👎 on desktop currently slides a phone-style bottom sheet with a second copy of the form over the form already on screen. Route to the visible panel instead. This is the "jarring, mobile-optimized" feeling.
Required field first, optional collapsed	M	high	Principle chips ("pick at least one") right under the thumbs; failure-kind and "what she should have done" tucked into "More detail (optional)".
Delete "What Clara said (quote)"	S	high	The pipeline already takes the quote from the pinned message (standing decision 08-23). The box asks graders to redo it by hand.
Delete "inference, not verified"	S	med	Zero readers anywhere in the pipeline.
Fix "What kind of failure?"	S	med	Only 3 options exist; stack them full-width so they stop reading as truncated. Widen the panel on large monitors (hard-capped at 320px today).
Fix "You disagreed"	S	med	It fires on any grader's disagreement, so it can be false. Say "you said 👎 · machine said 🟢" when it was you; neutral label otherwise.
Quiet the chip noise	S	low	Move "· new" (means unvetted, not recent) into the tooltip; collapse the add-a-chip box behind a "+".

Also on the card (found in review, not by the fleet)

The green check is sticky: stays on even after Clara sends more messages. Show "N new messages since" and count those as needing a look.
"Not yet graded" means you haven't — the machine color is deliberately hidden until you do. Say so; add a one-click peek.
A quality roll-up tile (KPIs above).

Decisions

Proposed — pending Fede's review. Each with a recommendation.

D1. Should Cerberus check real messages before Clara sends them?

A (recommended) — Shadow mode first: run it on every outgoing draft, log what it would have blocked, block nothing. Two weeks of data, then decide on blocking. Zero customer risk, real numbers.

B — Block immediately on the two highest-confidence checks (unbacked "I did X" claims, leaked internal reasoning). Faster protection, some risk of holding good messages.

C — Keep it CI-only. It stays a regression net, never a live guard.

D2. "You disagreed" badge

A (recommended) — Show "you said 👍 · machine said 🔴" only when it was really this grader; otherwise a neutral "Disagreement on file". Keeps the shared queue ranking identical for every viewer.

B — Just rename it "Needs another look" for everyone. Cheapest.

C — Personalize the whole queue per viewer. Reverses a documented invariant; bigger change.

D3. Initials: ship now or wait for full names?

A (recommended) — Ship "you / golden import / someone else" now; teammate initials as a fast follow once the name lookup is in the feed.

B — Wait until teammate names resolve end to end.

D4. Widening the desktop panel

A (recommended) — Add a wider large-monitor breakpoint on top of the existing laptop one. A previous attempt to edit the laptop values pushed the panel off-screen at 1024px.

B — Layout-only: stack the 3 failure buttons, leave the 320px cap.

D5. Collapse the "add a new principle" box?

A (recommended) — Behind a "+ New principle" click, auto-focused. One extra click to mint; removes a full-width control from every item.

B — Leave it visible; only fix the "· new" wording.

Rejected by the skeptics (for the record)

"Reuse the app's SegmentedControl for failure kind" — it's a strict required radio; failure kind is optional.
"Grader initials from the existing Avatar component are free" — identity is stored as an id, not a name; needs a lookup.
"Infer 'a teammate graded it' from graded-but-not-mine" — wrong whenever the row is a golden import (159 of them).
"The quote box has no consumers" — the anchor step does read it when present; the fix is deletion because the pin fallback already covers it, not because it's unread.

History

Decision record: ADR-0128. 35 commits Aug 11–26. Related docs: grading-playground-design, grading-annotation-guide, grading-desk-qa-2026-08-19, golden-dataset-practices, quality-loop-guide, cerebrus-burnin. Build sessions: Aug 11 (phases 0–1b), Aug 17 (main build), Aug 22 (right-click, failure classifier, blinding), Aug 24 (👎→test pipeline).