Grading Desk QA — Aug 19, 2026
Full QA pass triggered by Fede's report that the grader "ends at Aug 13" and feels sloppy. Ten investigation agents across two sweeps: pipeline coverage, scorer liveness, list ordering, a live write-path contract test, and a five-angle UI audit. Status: Shipped — all four fix batches merged Aug 19.
TL;DR — nothing is missing, and the scorer is keeping up. Every real conversation of the last 9 days (86 of 86) is present in the grading data, zero were eaten by the new test-traffic filter, and all 3 conversations that settled after the new scorer went live were machine-graded within ~21–90 minutes. The "stuck at Aug 13" look is the UI: the desk silently inherits whatever property was last picked anywhere in the app, sorts by flag severity with no date option, hides clean conversations from the default tab by design, and shows ~8 rows before the fold. Separately, the audit found 8 genuinely broken UI behaviors (worst: the grading pane can't scroll — 90% of long content is invisible), 2 real scoring-pipeline gaps, and one thing that deserves a business-level look: Yale has had zero inbound conversations of any kind since Aug 13.
Why the grader looked frozen at Aug 13
- The property picker follows you invisibly. The desk reuses the sitewide property selector (a saved cookie) with no visible indicator. If Yale was the last property picked anywhere, the grader shows only Yale — and Yale's last conversation of any kind is Aug 13, 19:59 UTC, matching the complaint to the second. (Best-fit inference — we couldn't inspect the actual browser session.)
- Severity outranks recency, always. Both tabs sort flagged-first; one red item from Aug 13 permanently owns row 1. The newest item sits ~40 rows down, and only ~8 rows fit on a laptop screen before scrolling. There is no "newest first" option.
- Clean conversations skip the default tab by design. "Needs a look" shows flags, unrated items, and a random 5-a-day sample of clean ones. Today's fresh, machine-green conversations only ever appear under "Everything" — nothing tells you that.
What checked out healthy
| Check | Result |
|---|---|
| Coverage: real conversations since Aug 10 present in grading data | 86 / 86 — zero missing; zero false positives from the Aug 19 test-traffic filter (147 excluded rows were all genuine bench/harness traffic) |
| Scorer liveness on post-deploy traffic | 3 / 3 graded, fastest ~21 min after settle (small sample — 15h of runway) |
| The "95 not rated yet" | Mostly history — 77% predate any grading mechanism existing; not scorer lag |
| Verdict write path (live test on the bench property) | Contract holds — 👎 with tag accepted; tag-less 👎 rejected (400); spoofed property/grader rejected (422); nonexistent item rejected (404); bench rows never surface in the real queue, graded or not |
Broken (fix first)
| # | Defect | Evidence |
|---|---|---|
| B1 | The grading pane cannot scroll: fixed 512px height with overflow:hidden. ~90% of a long checklist + "Clara's Summary" is simply invisible, with no scrollbar or cue. (Root cause of the previously filed scroll bug.) | Reproduced; 3,000px of scroll intent moved nothing |
| B2 | Middle pane collapses to 92px wide at 1024×640 — unreadable | Measured via layout rect |
| B3 | A failing verdict labeled "Unsuccessful" renders in success-green, directly above a red ✕ for the same result | Pixel-sampled colors |
| B4 | Reason-filter counts computed against the wrong scope — "Looked fine (40)" yields 0 results on the default tab | Live repro |
| B5 | Keyboard J/K moves the selection but never scrolls the list — selection walks off-screen | 15 keypresses, list never moved |
| B6 | 39% of queue rows show "Not rated yet" and a green "graded" checkmark simultaneously (imported human verdicts on machine-ungraded conversations) | 59 of 152 rows |
| B7 | A burst of internal test traffic (Aug 16, 03:46 UTC — six rows created within 160ms, one titled "Federico") sits in the queue as if it were resident activity | Queue data; same class as the fixed mailpit leak, different source |
| B8 | Score-section label clipped mid-word by the card edge ("…n 20/21 passed") | Screenshot |
Real pipeline gaps (not UI)
- degraded Automated follow-up nudges never trigger a re-grade. Clara's own cadence messages don't fire the settle event a real reply does — the conversation's timestamp keeps advancing while its grade stays frozen at the last human-triggered turn. Confirmed in 2 sampled conversations.
- degraded Suspected silent grade-save failure (one case, unconfirmed). One conversation's grading watermark advanced (the run executed) but no grade exists — consistent with the corpus write and the watermark living in separate, independently-swallowed error handlers. Needs a single read-only database check to confirm; the other 19 same-window unrated rows were not inspected.
- degraded No double-submit guard on verdicts. Posting the identical verdict twice creates two corpus rows silently (append-only is intentional; the accidental-double-click path isn't).
Yale silence — resolved
Yale has had zero conversations since Aug 13, 19:59 UTC. Resolved 2026-08-19 (Fede): Yale is not live yet — the silence is expected. No action. The finding stands only as the explanation for the frozen-looking grader when the desk is scoped to Yale.
Confusing / sloppy (the "death by papercuts" list)
| Area | Defects |
|---|---|
| Finding things | Search matches display-name only (not email/phone), exact-substring, no typo tolerance, silently scoped to the open tab; search text survives switching item types and shows a misleading "no results"; no deep link into an item; no "Grade this" button from a conversation or prospect page; reload loses selection; tab + filters reset on every reload |
| Ordering & freshness | No date sort anywhere; 3-month-old rows resurface unlabeled (side effect of the "show graded items regardless of age" fix); dates never show the year; "unrated" count mixes pre-scorer history with live work |
| Row quality | 25% of rows titled with a raw phone number ("Search by name" placeholder notwithstanding); one caller appears as 9 separate unlinked rows; visually identical duplicate rows; malformed name casing rendered as-is; vendors/internal people treated identically to residents |
| Pane behavior | Re-selecting a viewed item refetches with a full skeleton; old rows stay clickable under the "Loading…" state; "0 of 0" flashes on load; Escape/click-away can't deselect; switching sub-tab silently drops selection; hover tooltip covers the text it explains; no "more below" cue on clipped content |
| Copy & polish | "Filter filter by reason…" placeholder; duplicated empty-state copy with two wordings; "Work Order Id"; header truncates a name that fits one column over; mismatched thumb icon sizes on the two surfaces meant to look identical; dead space in verdict pane; empty-queue J/K hint; empty-state alignment differs between panes |
Fix plan — ALL FOUR BATCHES SHIPPED same day (Aug 19)
Status: every batch below merged to production on Aug 19 after multi-round review. The desk is now named Quality Desk. Residual open items: the internal-tester phone setting (awaiting Fede's go), the repeated-identical-nudge blind spot (carded), the renewal-conversations-never-grade question (carded), and whether the desk should default to All Properties (flagged in the findability PR).
- Batch 1 — broken (ship now): B1–B8. Mechanical, low-risk, high-visibility.
- Batch 2 — findability: search across name+email+phone with normalization, cross-tab hits, deep link (
?item=), "Grade this" button on conversation/prospect pages, visible property-scope indicator + default to All Properties on the desk. - Batch 3 — freshness: newest-first sort toggle, "up to date through <time>" banner, year on old dates, label resurfaced old rows, split the unrated count (historical vs live).
- Batch 4 — pipeline: cadence-touch settle signal, atomic grade-save + watermark (confirm the suspected silent miss first with one DB read), double-submit guard.
Provenance & honest caveats
Two workflows, 10 agents, 2026-08-19. Write-path test performed live against the bench property only (two test verdicts left there, ids in session logs). The cross-examiner flagged two internal count inconsistencies (a Camellia-scoped count exceeding the all-properties count, and row positions ~25–70 earlier than pure bucket math predicts — likely an undocumented graded-rows-sink rule) — treat exact positions as approximate; the coverage and liveness conclusions are unaffected. Not checked: Fede's actual browser session state, client-side caching, whether severity-first ordering is recent or long-standing, and 19 of 22 in-window unrated rows.