ADR-0095 — Universal conversation classification: topic · subtopic · regarding, nothing ever empty

Amendment (2026-08-28) — a narrow, bounded exception to "stamp once, never overwrite": owner call, verbatim: "as new messages come in we can update previous ones… we don't want to wait for it to be perfect the first time, we'd rather be optimistic and fix it as more context arrives." applyTopicTags (message-topic-classifier.ts) gained a REVISION branch: a TENANT-authored, untied (!regardingId), non-sticky message within the last 4 classifiable messages of the thread (not reaching back across a >24h gap) may have its REAL topic replaced by a different real topic when later context changes the maestro's read — topic, label, and subtopic all move together so no stale value survives under the wrong topic. Every other message — outside the window, Clara-authored, entity-tied, or older than the recency cutoff — keeps the original never-overwrite guarantee exactly as this ADR specifies. See selectRevisableIndexes (conversation-classification.ts) for the exact eligibility rule and its reasoning (a Clara/assistant row can carry a deterministic stamp from a tool-call tie or the vendor-callback recognizer with no provenance flag distinguishing it from a maestro guess, so it is never offered for revision).

• Status: Accepted (owner sign-off 2026-07-15)
• Date: 2026-07-15
• Deciders: Gera (owner); design consolidated from the decision-complete plan at ~/code/tmp/message-classification-unification-plan.md (grounded on origin/main 2026-07-15; all six open decisions resolved + owner-signed there).
• Numbering note: the planning doc calls this "ADR-0090" — that number was a placeholder minted before ADR-0090–0094 landed. This ADR supersedes the ADR-0090 placeholder name in the planning doc; the content is the same.
• Relates:
  • ADR-0055 — the topic taxonomy. This ADR explicitly supersedes ADR-0055 §"The abstain rule (no general/other bucket)": the classifier becomes forced-choice, and the honest catch-all none (UI label "No topic") replaces abstention. The taxonomy itself, the boundary rules, and the ban on general/other junk drawers all stand.
  • ADR-0073 — topic + subtopic model. Amended: subtopic grows a per-topic vocabulary registry (SUBTOPIC_VOCAB) and a message-level field; the conversation-level roll-up keeps its ADR-0079 D3 stamped-not-derived semantics.
  • ADR-0083 — topic-scoped regardingId/regardingLabel. Amended: the pointer gains an explicit, self-describing regardingKind so consumers stop inferring the entity table from the topic (the PR #3293 bug class dies by construction).
  • ADR-0079 — single writer. Unchanged and load-bearing: every classification write keeps flowing through the one planning module + the one writer sink.
  • ADR-0084 / ADR-0088 — trigger topology. Unchanged: the bridge registry's existing consumer (now classification-stamp, renamed from topic-stamp in PR-6) + the 15-min reconciler remain the only triggers; no new consumer, no parallel path.

Context

Message/conversation classification today is a sprawl of overlapping fields with three different vocabularies presented as sibling rows in the message popup ("Type" = kind, "Topic" = regardingType, "Related " = regardingLabel), a maintenance-only subtopic, and — the operator-visible failure — blank categories: formatConversationCategory renders "" for an unclassified thread, so the /conversations list carries uncategorized rows that can't be filtered, excluded, or trusted.

The 2026-07-14 backtest (~700 prod conversations) measured the blanks: ~20% of conversations, 95% voice, concentrated on a single voice property. Their composition: ≈38% silent hang-up (zero caller speech), ≈30% dead air after engagement, ≈15% wrong target, ≈3% spam, and only ~7% a genuinely missed real topic. ADR-0055's abstain rule — correct as a defense against junk drawers — made this noise indistinguishable from missing data and made "exclude spam" filtering impossible.

The owner's goal: one schema every message shares — topic + subtopic + relevant resource — with a hard guarantee that nothing is ever empty, a full backfill of history, and single-writer discipline.

Decision

1. The unified classification triple (message = canonical, conversation = roll-up)

Every message carries ONE classification shape:

interface Message {
  /** What this message is about — SINGULAR dominant domain topic.
   *  (Today `regardingType`; RENAMED to `topic` in PR-5.)
   *  Enum: the 11 existing values + 'none'. Required post-backfill epoch. */
  topic?: ConversationTopic;
  /** Finer key under the topic, from SUBTOPIC_VOCAB. NEW at message level.
   *  null = "this topic has no finer key here" — honest, visible empty. */
  subtopic?: string | null;
  /** The relevant resource — explicit, self-describing entity pointer.
   *  `kind` is NEW (ADR-0083 made explicit); id/label are the existing
   *  fields. Storage attrs: regardingKind / regardingId / regardingLabel. */
  regarding?: { kind: RegardingEntityKind; id: string; label?: string };
}

The conversation carries only derived/stamped roll-ups — topics/topicsAllTime (derivation unchanged except the none-eviction rule, §4) and the vocabulary-validated subtopic. No new conversation fields. formatConversationCategory keeps rendering Topic · Subtopic; post-backfill it can never be "" again.

Message.kind is provenance/format, not classification — untouched at write time (the "no anonymous outbound" schema and the ~60-value chip system depend on it), removed from the popup's classification block and regrouped under a provenance block ("Sent as", PR-3). triggerSource/triggerActor/author/writerId are explicitly out of scope.

2. Forced choice — supersedes ADR-0055's abstain rule; the none catch-all

The classifier becomes forced-choice: every classifiable message gets a topic; a message with no property-management matter gets the catch-all.

Enum token none, UI label "No topic" (owner-approved). Honest about the message ("there is nothing to label"), not about us — expressly NOT unknown (implies we failed) and NOT other/general (a real topic that missed the list must surface as a taxonomy gap, not hide in a drawer; the taxonomy drift test bans all three tokens forever). Short, greppable (isRealTopic(t) centralizes t !== 'none' in topic-labels.ts), and a plain string in YAML/JSON eval datasets, unlike null.

Junk-drawer mitigation (the reason ADR-0055 abstained): (a) none is self-describing — its subtopics say WHY nothing was labeled; (b) the eval golden set gains cases where none is FORBIDDEN as well as required, so over-tagging into none is a red eval; (c) a weekly %none ops counter makes upward drift a taxonomy-gap alarm.

3. The five dispositions — none's subtopics

Precedence ladder (top wins): 1. silent_hangup (deterministic — fires before the LLM ever runs) → 2. wrong_number/spam (explicit signal beats shape) → 3. ghosted (engagement started then died) → 4. unrelated (real content, not our domain).

The ghosted scope rule: voice dead-air ONLY. An outbound SMS/cadence that never gets a reply keeps its real kind-derived topic (a tour reminder stays tour) — non-reply is an engagement fact, not aboutness, and those outbound rows are already stamped at append time by deriveTopicFromKind (outbound-log.ts; named deriveRegardingTypeFromKind until the PR-6 identifier rename). none can only ever be assigned to a thread whose inbound content carries no PM matter.

The set is the backtest's empirical partition (38+30+15+3 = 86% of blanks in four legible buckets; ~7% genuine must land in real topics under forced choice, eval-pinned). Dispositions are not sticky; adding/removing one is a one-line SUBTOPIC_VOCAB change + a targeted re-stamp.

4. Overwriteability + spam-stickiness safety (three coordinated rules)

1. Message level: the never-overwrite guard (applyTopicTags) is relaxed for exactly one value — a stored none may be UPGRADED to a real topic. Real topics remain immutable (ADR-0055/0079 discipline unchanged; none is definitionally the absence of a label, so upgrade-only mutation preserves the invariant's intent).
2. Derivation level (the real spam-stickiness fix): deriveConversationTopicSets gains the none-eviction rule — none appears in topics/topicsAllTime ONLY when no real topic qualifies for that set. A reused row (repeat caller) that was all noise last week and is a maintenance request today derives topics=[maintenance] automatically, with zero overwrites of message history. none is emphatically NOT in STICKY_TOPICS.
3. Subtopic level: a none-vocabulary subtopic (e.g. spam) on the conversation is cleared/replaced whenever the derived topic set gains a real topic — so "Maintenance · Spam" chimeras cannot render.

5. Subtopic vocabulary registry — SUBTOPIC_VOCAB

A total Record<ConversationTopic, readonly string[]> in topic-labels.ts (tsc forces a decision per topic; a drift test pins content). Ships with exactly two populated vocabularies — maintenance (the existing 13 playbook keys, VALID_ISSUE_TYPES reused verbatim) and none (the 5 dispositions). Every other topic is explicitly [] → subtopic: null, rendered as an honest "—". Per-topic vocabularies are deferred to calibration fast-follows (propose → calibration batch → eval pin → ship); inventing them now is how junk taxonomies are born (ADR-0055's own history). A per-topic general default subtopic is rejected — it reintroduces the junk drawer one level down. Dispositions render through DISPOSITION_LABELS plain-English copy (ghosted → "Dead air", owner-approved) — no enum strings in customer-facing panels.

Amendment (owner-confirmed 2026-07-18) — maintenance coarse trade buckets, a read-side LAYER over the playbook keys. The 13 playbook keys are the right stamping granularity but too fine for the /conversations list's filter chips, so topic-labels.ts adds MAINTENANCE_BUCKETS = plumbing | hvac | electrical | appliance plus a total MAINTENANCE_SUBTOPIC_BUCKET: Record<IssueType, MaintenanceBucket | null> map and a maintenanceBucketOf() resolver. The evidence earning this vocabulary (per this section's own rule) is the list-page filter need. Decisions inside the amendment: layer over, never replace — the stamped vocabulary stays VALID_ISSUE_TYPES verbatim (the playbook stamper, the self-serve WO gate, and issueTypeToCategory all keep consuming the fine keys; nothing about the write path changes); general stays banned at the bucket level too, so window_stuck (carpentry/handyman) is honestly bucket-less (null) rather than forced into a trade or a junk drawer; smoke_detector_beeping → electrical, deliberately diverging from issueTypeToCategory's general WO category (the two maps serve different systems — vendor-dispatch category vs. read-side list grouping — and the divergence is documented at the map). Consumer: the /conversations list's "Issue" multi-filter (matchesMaintenanceBucketFilter), which mirrors formatConversationCategory's coherence guard so a chimera row never matches a chip its visible category contradicts. Content pinned by conversation-subtopic-vocab.drift.test.ts.

6. Explicit regarding.kind

The resource pointer gains a self-describing kind. Enum = the EXISTING 8-value RegardingEntityKind union (work_order | prospect | tour | renewal | turnover | charge | lease | vendor) reused verbatim — no new values. Storage attribute regardingKind. The ADR-0083 incident is the argument: PR #3293 stamped regardingId: tour.id and broke three consumers that all assumed bare ids were WO references. The explicit kind kills the bug class (a consumer checks kind === 'work_order', full stop) and unlocks cross-kind ties the topic-scoped map structurally forbids. REGARDING_ENTITY_KIND is demoted to migration-only in PR-5 and deleted in PR-6 (it remains the write-time default for topic-tied stampers, folded into the writer).

7. Full DDB rename (later PR — decided now)

regardingType→topic (+ regardingKind) is a FULL storage-attribute rename, via the proven lazy read-adapter (tenantName→participantName pattern) PLUS an explicit convergence sweep. TS-surface-only is rejected (a permanent name split = a forever translation layer = a legacy path by definition); lazy-only is insufficient here because MSG# rows are append-only and never re-saved (preserveUpdatedAt), so without the sweep the old attribute — and the adapter — live forever. The sweep costs no extra table pass: it rides the same per-row UpdateItem the re-derivation + regardingKind stamp already require. Adapter-first ordering makes the transition order-safe; PR-6 deletes the adapter, restoring grep-zero.

8. Census-gated backfill

Six ordered, individually idempotent, dry-run-first sweeps (census → issueType residue removal → deterministic stamps → LLM classification → conversation re-derivation + rename convergence → audit), per-property, test properties excluded, ADR-0073-style run ledger, no message content in any log (ADR-0026). Deterministic before LLM: kind-mapped outbound rows are stamped free by importing the EXISTING runtime derivation deriveTopicFromKind (deriveRegardingTypeFromKind pre-PR-6; one source, no script-local map); zero-caller-turn voice threads get none·silent_hangup deterministically.

The paid step is gated: the Step-0 census number is posted first, and Step 3 spends nothing until the owner says "go" — the script refuses to start without a --go flag and prints projected spend from the census. Ceiling: $100 hard / 5,000 paid conversations (expected spend ≈ $5–50; exceeding the ceiling means the census surprised us, which deserves a human look, not an auto-spend). Same prod classifier path + tier (AGENT_MODEL, temp 0) — no cheaper-model fork: history must be stamped by the same brain that stamps the present. Stamps are first-stamp-wins + runId-ledgered, so a bad run is surgically un-stampable.

9. Single-writer wiring

ONE domain module owns every classification write — conversation-classification.ts (named conversation-topic-stamping.ts until the PR-6 rename) — with applyTopicStampPlan + the vocabulary-validated successor of stampMaintenanceSubtopic in conversation-writer.ts as the sole DDB sink. Trigger topology is unchanged: the bridge registry's existing classification-stamp consumer (lambda/conversation-events-bridge/registry.ts, named topic-stamp until the PR-6 rename — one honest-naming moment; the feared eventId-prefix blip turned out not to exist: the consumer name feeds only the CloudWatch failure-log label, signal eventId prefixes are hardcoded), the 15-min reconciler (now also the ghosted/disposition finisher for threads with no terminal signal), the structural stampers (call-ended silent-hangup, playbook subtopic, tenant_confirmation, emergency — all calls INTO the module), and the backfill scripts. No new SQS consumer, no second workflow, no classify call outside the module, no UI-side inference.

"Nothing is ever empty" enforcement layers: (1) forced-choice classifier — no abstain output accepted; (2) terminal-turn structural stamp for content-free voice; (3) the reconciler sweeps anything missed within ≤15 min; (4) a post-epoch runtime detector (CLASSIFICATION_EPOCH, mirroring the SINGLE_WRITER_EPOCH pattern) alarms on any surviving blank. A brand-new conversation is legitimately blank for ~2–15 minutes; the UI renders that window as "Classifying…", not "—" (PR-3).

10. Default list behavior for none·spam

Spam threads stay VISIBLE by default in /conversations, wearing the "No topic · Spam" chip, with a one-click persistent "Hide spam" filter (owner-approved). Never-lose-a-real-caller beats feed cleanliness while the classifier's spam precision is unproven; the volume (≈3% of blanks ≪ 1% of threads) doesn't pay for the burial risk. Revisit trigger: if spam exceeds ~5% of new threads AND the eval shows sustained spam precision over several weeks, flip the default — a one-line change either way.

Drift guards + evals

PR sequence

Ordering rationale: writers (PR-2) before backfill (PR-4) so history is stamped by the same brain that stamps the present; UI (PR-3) can ship between them; the rename (PR-5) is deliberately after the backfill so the sweep and the rename converge in one storage pass per row.

Consequences

• Positive: every message and thread is classifiable, filterable, and honest — blanks become legible dispositions instead of missing data; the popup's three vocabularies collapse into classification vs provenance; the pointer self-describes; per-topic vocabularies become one-line, drift-guarded additions; single-writer + single-trigger discipline is preserved end to end.
• Negative / accepted: forced choice slightly raises misclassification risk on ambiguous free-text (the blank was ALSO a wrong answer — it just looked like missing data); most topics render "· —" until calibration batches earn vocabularies (the visible empty is a deliberate validation aid); message-level subtopic is new write surface for the maestro + playbook stamper; historical conversation subtopics cannot be pushed down to specific messages (accepted — forward-looking, plus none dispositions which ARE call-shaped).
• Reversal notes: the enum token is locked now (cheap pre-backfill, a full sweep after); the UI labels stay one-line copy changes forever; dispositions are cheap to revise; the storage rename is hard to reverse after the sweep — which is exactly why the adapter-first ordering is mandatory.
• This ADR deliberately reverses the 2026-07-14 backtest session's disposition-as-third-axis proposal (owner decision): disposition folds INTO subtopic under the catch-all topic — two levels, no third axis.

Amendment — 2026-07-23 (owner decision): enrichment + eager blasts + announcement

Three related gaps closed in one pass (the "why is this still Classifying… / Subtopic —" session):

1. Subtopic enrichment on deterministic-topic rows. applyTopicTags previously skipped any row already holding a real topic — so a row whose topic was stamped deterministically at write time (outbound kind→topic derivation) could never gain a finer key, and deterministic-only threads read "Subtopic —" forever. Now such a row keeps its topic/label provenance untouched but may EARN a subtopic when the maestro agrees on the topic (tag.topic === row.topic — vocabulary is topic-scoped) and the row has none. The maestro is also triggered for enrichment-only work (subtopicEnrichableCount), bounded to threads whose conversation-level subtopic is still unset so a null-returning thread doesn't re-spend an LLM call per settle forever.
2. Eager classification of mass-comms blasts. A blast-only thread never sees a settled turn, so nothing ever requested classification for it (ADR-0094 deleted the sweep). The blast fan-out (mass-comms.ts) now signals signalUntaggedMessagesArrived directly per appended recipient thread — NOT via a conversation.turn_settled event, which would fan out to the grade + maintenance-judge consumers (per-recipient LLM workflows with nothing to judge). Sanctioned second caller in turn-settled-single-path.drift.test.ts; the settle path itself stays single-path.
3. announcement — the first maestro-classifiable maintenance subtopic. SUBTOPIC_VOCAB.maintenance = the 13 playbook keys + MAINTENANCE_TEXT_SUBTOPICS (['announcement'] — property-wide maintenance/utility notices, typically mass-comms blasts, e.g. a water-shutoff notice → maintenance · announcement). isMaintenanceSubtopic narrows to VALID_ISSUE_TYPES only, so the self-serve WO gate, the trade-bucket layer, and the planner's never-displace guard keep playbook-key semantics — a text-classified announcement can never mint a self-serve WO or masquerade as playbook provenance. maestroSubtopicVocab('maintenance') offers exactly the text keys, never the playbook keys.