A design for Cerebrus and Agent Smith grading what happens in the apartment channels — every layer of a turn, not just the reply, and attached where it can actually be argued with.
2026-09-02 · draft for discussion · nothing built yet
Most of this already exists. Cerebrus grades whole threads today, GradeVerdict already captures a human 👍/👎, and the D8 nightly sweep already turns a 👎 into a Cerebrus fixture — it ran successfully today. What is missing is the Slack surface, not the machinery.
The instinct that "Cerebrus already is a system that could grade stuff" is right, and it goes further than that — the disagreement loop is built too. Only the last row is new work.
| Capability | Where it lives | Status |
|---|---|---|
| Grade a whole thread, not just one reply | cerebrus/src/conversationEngine.ts | exists |
| A 4-value verdict vocabulary | pass · repair(span) · hold · block | exists |
| Capture a human 👍/👎 plus principle chips | GradeVerdict (ADR-0128) | exists |
| Turn a 👎 into a regression fixture, nightly | D8 pipeline sweep — 6 of Cerebrus's 30 cases came from it | exists |
| Post narrative updates into apartment channels | Agent Smith | exists |
| Do any of it in Slack, where people actually are | — | missing |
The instinct to reach for 1–10 or 1–5 is the common one, and both the literature and our own measurements point away from it.
What the research says. Binary criteria yield the highest inter-rater reliability, because there are only two categories to distinguish. Likert scales' inherent subjectivity limits reproducibility, and decomposing a multi-dimensional Likert rubric into fine-grained binary criteria measurably improves agreement. Where a graded scale is genuinely wanted, the recommended ceiling is 0–3 or 0–4 — not 1–10.
There is a second problem with a scalar, independent of noise: it gives nobody anything to do. "4/10" does not say what to fix. missed_followup: FAIL does.
worstState.pass · repair · hold · block — for the deterministic slice. It is reproducible by construction: the harness makes zero LLM calls.So a post leads with BLOCK · unbacked promise, not 4/10.
This argues against a single composite score for a conversation. Per-dimension numbers, each with a stated reason, are a different question and a better idea — see §6.
Concise by default. The dimensions are the payload; the prose is one line.
Illustrative. The verdict, dimensions and reason above are the real shape of a Cerebrus conversation verdict; the message text is mocked.
Rating only what Clara said is rating the last inch of a long pipe. The interesting failures happen upstream: she was handed the wrong facts, or she never called the tool that would have told her, or she looped. A verdict that says "the reply was bad" without saying which layer was bad sends an engineer to read a prompt when the real fault was an empty context block.
This is not a "we'd need to start logging" project. Every claude_call step on an AgentTrace already persists:
| Already stored per turn | Makes gradeable |
|---|---|
| The full system prompt as sent to Claude | Context — was the fact needed to answer actually in the block? |
loopIteration | Loop — how many passes, and did they make progress? |
tool_execution steps with status | Tools — were the mandated tools called? did one error and get quoted as policy? |
stopReason, tokens, cache reads | Shape — truncation, runaway cost |
| The draft itself | Output — tone, promises, numbers |
There is already a viewer at admin/dev/agent-traces and a per-conversation trace endpoint. The grading surface would read what the trace tab already shows.
"Context: 4, because we gave it too little context" is not a criticism of Clara. It is a criticism of us. That distinction is the most useful thing in the whole scheme, and it should be structural rather than left for a reader to infer.
tone · rude, cold, or cheerful about a complaint
pacing · replied when the conversation was over
output · invented a policy, quoted a wrong number
loop · didn't call a mandated tool, or looped without progress
A 👎 here becomes a Clara fixture. The fix is a prompt change and it ships behind the gate.
context · the fact needed to answer wasn't in the block
tools · no tool exists that could have answered
truth · the ledger itself was stale or wrong
A 👎 here is not a prompt bug and no prompt change fixes it. It becomes an integration or data ticket. Filing it as a Clara failure buries it.
This matters more than it looks. The replay found that 75% of failures were missing-fact — Clara not knowing something. Graded as "Clara was bad" that is a demoralising, unactionable number. Graded as "our context was thin" it is a roadmap. Same rows, different queue.
Earlier this page argued against a number. That argument was about a single composite score for a conversation, and it stands. Per-dimension numbers with a stated reason are a different and much better thing — but the noise still binds, so the two kinds of dimension get different treatment.
| Kind | Scale | Why |
|---|---|---|
| Deterministic tool gate, fact-present, loop count, numeric match | PASS / FAIL | Computable from the trace with no model call, so it is exactly reproducible. A number here would only add false precision to something already certain. |
| Judged tone, pacing, helpfulness | 0–3 | The research ceiling for a rubric scale. At a measured 25% label churn you cannot honestly distinguish a 6 from an 8 on a 10-point scale — but "harmful / poor / minor / fine" survives that noise. |
Never average across the two. A deterministic FAIL and a judged 3 do not combine into anything meaningful, and the moment they are averaged, adding easy deterministic checks starts inflating the score.
The real tension: one Slack message per Clara turn is precise but floods the channel; one post with a table of turns is compact but you cannot react to a table row. Numbering the rows and replying "message 4 is bad because…" solves it on paper and fails in practice — free text has to be parsed, and a reference to "message 4" breaks the moment anyone edits the post.
One post per turn in-channel — a 6-turn conversation is 6 posts. The channel becomes unusable and everyone mutes it.
One post, numbered table, reply with references — "msg 4 bad because ABC" is free text someone has to parse, it breaks on an edit, and two people replying at once produce ambiguity about which row.
Parent = conversation verdict, posted only if a turn was flagged. Worst dimension in the header.
Thread reply = one flagged turn, carrying its own dimensions. 👎 on that reply targets that turn, structurally. A written reply under it is a comment on that turn and needs no reference at all.
Passing turns get nothing — most turns pass, and a thread of "fine" is the same noise problem one level down.
Illustrative. The dimensions and the two-kinds split are the proposal; the message is mocked. Note the header verdict names the output failure, but the row an engineer needs is context — which is why both are shown rather than one summary.
"She replied when the conversation was already over" cannot be judged from one turn in isolation — it is a property of the thread. Same for "never followed up" and "the escalation never closed". Forcing every dimension to the turn level would make these unanswerable; forcing them all to the conversation level loses the precision that makes a 👎 actionable.
So dimensions carry a scope, and the parent post shows the conversation-scoped ones while each thread reply shows the turn-scoped ones. pacing appears on the turn that shouldn't have happened; followup_exists appears only on the parent.
Four pairs. Each left-hand version is a plausible thing to build and each one fails for a different reason.
Nothing here is actionable. 6 of what? Which part was bad? And on a re-run this is a 4 or an 8 — the same conversation, a different number.
Names the failing dimension. An engineer knows which rule fired and where to look before opening anything.
A conversation that invented a policy is not 75% fine. Averaging lets one serious failure be outvoted by three trivial passes, and the number goes up when you add easy checks.
Worst dimension is the headline; the tally is context, never the verdict. Adding more easy checks cannot improve the grade.
This conversation may have quoted the wrong rent, been rude, or over-promised — none of which any deterministic predicate looks at. "PASS" invites everyone to stop reading.
States its own scope. A reader can tell the difference between "we looked and it was fine" and "we didn't look".
A post per conversation, all day, most of them PASS.
The channel becomes unreadable within a week and people mute it — which silently disables the whole system, because the loop depends on a human noticing.
Post non-pass verdicts only, plus one daily line: 47 graded · 44 pass · 3 below.
The count proves the grader ran — a silent channel could otherwise mean "all good" or "the job died", and those must not look alike.
This is the valuable half, and it is well-founded: mining disagreement cases creates a natural priority queue for improving prompts and for focusing expensive human attention. Four of these five steps already exist.
The only new pieces are a Slack event handler and a write into an existing store. No new pipeline, no second grading system, no parallel source of truth.
A 👎 is only worth collecting if it can become a test. The difference is whether the reply says what the right answer was.
👎 “this is wrong”
👎 “bad grade”
👎 “Clara was fine here”
No expected disposition, so nothing can be pinned. D8 would capture a row that no fixture can be built from, and it sits in the queue forever.
👎 “Should be PASS — the arrival section does exist for Lampliter, so the promise was keepable. The grader read the wrong property.”
Names the expected verdict AND the reason. That is a fixture: same input, expected: pass, and it fails until the grader reads the right property.
Two different things arrive through the same 👎, and they must not be conflated:
This one is real and shipped today, which makes it a fair test of the design. Nothing below is hypothetical except the grading steps themselves.
The defect. Clara's tour confirmation said "We'll text you access instructions about an hour before your tour." The T-1h reminder always sends, but carries arrival instructions only when the property has them on file — and renders silently without them when it doesn't. Across one client's 13 properties, exactly one had that content. So on 12 of 13 properties the promise could not be kept.
| Step | What the system would produce |
|---|---|
| 1 · Verdict | BLOCK · unbacked promise — no_ungrounded_policy_claim fails: the draft states a future action the system cannot guarantee. The other two predicates pass. |
| 2 · Post | Lands in the property channel because it is not pass. One line of reason, links to the conversation and to the rule that fired. |
| 3 · Human | Reacts 👍 — the grade is correct. No calibration case needed; this is a real defect. |
| 4 · Fixture | The nightly sweep captures it: the draft, the property context showing no arrival section, expected: block. |
| 5 · Fix | The prompt stops promising contents it cannot guarantee. The fixture flips to passing, and the gate holds the line. |
What actually happened instead: this was found by hand, in an eval-coverage audit, after it had been live long enough to be the largest single contributor to the hallucinated-policy rows in a client's conversation replay. It shipped as PR #6812. The system proposed here is the difference between finding that in an audit and finding it the first afternoon it happened.
That is the honest boundary. "Cerebrus grades the apartment channels" is true for a narrow, reproducible slice and misleading as a general claim. Everything past those three predicates needs the LLM judge lane — which is exactly the part carrying the 25% churn.
Real failure classes from the graded corpus, against what a deterministic grader would actually catch.
| Failure | Example | Caught by |
|---|---|---|
| Never followed up | Resident asks a question, thread dies | followup_exists |
| Escalation never closed | Forwarded to a PM, no resolution ever recorded | loop_closed |
| Invented a policy | "We'll text you access instructions" with none on file | no_ungrounded_policy_claim |
| Quoted the wrong rent | $1,950 when the ledger says $1,850 | turn-level only |
| Answered in the wrong language | Resident writes Spanish, Clara replies English | nothing |
| Tone | Curt, or cheerful about a complaint | nothing |
| Over-promised | "I'll have someone there this afternoon" | nothing |
| Blast buried a live question | A property-wide broadcast is the only "reply" | nothing |
Five of eight fall outside the conversation engine entirely. That ratio is the design constraint, and it is why every post has to state its own scope.
The practical consequence: ship the deterministic slice first and label it as such. A grade that is silent about what it did not check is worse than no grade, because it reads as a clean bill of health.
Each of these has a specific, checkable early symptom. Worth agreeing them before building, because each is invisible from inside the system.
| Failure mode | What it looks like | Early symptom to watch |
|---|---|---|
| Alert fatigue | The channel fills with grades, people mute it, and the loop silently stops — the grader keeps running and nobody reads it. | Reaction rate per post falls. If nobody has reacted in a week, the system is off regardless of what the logs say. |
| Grade inflation | Adding easy dimensions raises the pass rate without changing behaviour. Especially tempting if anyone reports the rate upward. | Pass rate rising while the 👎 count is flat. Never report a composite percentage as a health metric. |
| Right answer, wrong question | The check is reproducible and confidently measures the wrong thing — exactly what the promise tracker does today. | A dimension that has never failed. A predicate at 100% for a month is either unnecessary or broken. |
| Judge drift | The judged dimensions move because the model changed, not because Clara did. | Re-grade a frozen sample on a schedule. If yesterday's conversations grade differently today, the instrument moved. |
| Calibration debt | "Clara was fine, the grade was wrong" 👎s pile up unfixed; people stop trusting the grades and stop reacting. | Track grader-error 👎s separately from Clara-error 👎s. If the first is growing, stop shipping and fix the grader. |
| Scope creep into people | The grader starts scoring what a colleague wrote in a property channel. | Any grade whose subject is not a Clara outbound. Make that structurally impossible rather than a rule. |
#apartment-camellia came back with empty text through the Slack API, on both concise and detailed reads. If that is the known block-rendering behaviour, a grader reading via the API sees nothing. Worth confirming before designing around it.apartment-*" includes calls, texts, bookings and human chatter. Grading a colleague's message is a different product from grading Clara's. Recommend: Clara's outbound only, at least to start.GradeVerdict.graderUserId is a foreign key to a real User. Slack identities must map onto that, or the provenance of a verdict gets weaker exactly where it matters.pass rows is the likelier shape.Build the surface, not the system. Wire Smith to read Cerebrus's verdicts and the AgentTrace that already sits behind every turn. Post one parent per conversation into the property channel, only when a turn is flagged, and one thread reply per flagged turn carrying its own dimensions. Let a 👎 on a thread reply write the GradeVerdict row the nightly sweep already consumes.
Grade every layer, and say which one failed. Context, loop and output are separate rows, and each is marked as ours or Clara's — because a thin context block and an invented policy are different bugs with different owners, and merging them into one score buries the more common of the two.
Start where it is reproducible. The deterministic dimensions — was the mandated tool called, was the fact in the block, do the numbers match the ledger — need no model call and cannot drift. Add judged dimensions (tone, pacing) one at a time, each on 0–3, and only let one colour a post after its agreement with a human has been measured against the 0–3 point noise floor.
Explicitly do not: invent a 1–10 score, average across deterministic and judged dimensions, post a passing turn, or let a PASS go out without naming what it did not check.
cerebrus/src/conversationTypes.ts · ADR-0128 (D8)