0070 — An alert class has one printed ID, and its fix history is derived from the PRs and the ledger, not kept
- Status: Accepted 2026-10-20 (proposed 2026-10-19, ADR-0004). Built: propflowai #21292 (the printed ID, live in prod 2026-10-20), #21319 (
automation_run_failureclassed by job), #21317 (CI cards print and stamp it); agent-smith #1274 (the history lookup, ladder and prompt; merged as0ed9cfea, socket and worker booted on it 2026-10-20T03:56Z). Still to build: the nightly engine's history (part 4), Pythonnotify()parity (part 2), and the post-fix recurrence dedup bypass on a durable first-sighting record (part 1c). Build plan: agent-smithdocs/planning/alert-ids.md. - Date: 2026-10-19
- Deciders: Gera asked for it (2026-10-19): "give a type of alert or bug an ID somehow, so if we see that same alert again we can search on Slack: oh we've seen this alert before, we tried to fix it this way, that fix didn't work, let's think bigger."
- Systems: propflowai
notify()/sendBakeAlertand the CI workflows that hand-postsmith_alertcards; Agent Smith's Pythonnotify(), its socket router (_route_alert) and both fix engines (the alert decider of ADR-0056 script-fix-pr and the nightlyjob-failureremediation); the outbound ledger.
The problem
Smith fixes an alert, the alert comes back, and the next run fixes it again as if for the first time.
identity_contact_match_refusedgot seven merged Smith fix PRs between 10-01 and 10-16, four of them on one day. The class re-fired after each merge. No PR said whether the one before it held.maintenance-eval-dailygot 20 merged PRs, most of them re-describing "the same steady-state cluster".- An audit of the 69 alert-fix PRs merged 10-05 to 10-19 classed 13 as root fixes and 43 as bandages; 28 of the 43 bandaged alerts fired again.
- Two seats (the Gera and Fede logins) opened about ten duplicate fix PRs for alerts Smith was already fixing.
Three mechanisms make this the default rather than bad luck. The decider's resume-search reads only OPEN PRs on its branch prefix, so a merged attempt is invisible to the next run. A re-fire within 48h is skipped as a duplicate of the earlier verdict, so a recurrence after a fix is never looked at as a recurrence. And a person asking Slack "have we seen this?" has nothing to search for, because the class name lives in message metadata, not text.
What already existed (most of it)
- A stable class key, used everywhere except the text.
source: notify.tsalertSource, every bake-alertkind, the Pythonnotify(alert_source=), the CI cards'event_payload.source. Smith's router,alert_signature,error_class, the ignore rules, the ledger'sforeign:<source>rows and thesmith-fix/<kind>-<source>branch prefix all key on it. - Attempts are recorded. Every Smith fix is a PR on a deterministic branch prefix, and GitHub holds its number, author, merge time and sha.
- Occurrences and verdicts are recorded. The ledger has a row per app/CI alert post (
foreign:<source>, via the daily reconcile) and per Smith post, verdicts included.recent_alerts.note_repeatcounts re-fires inside the dedup window. - A history file exists for one surface. failure-notes.md is edited by hand through PRs, covers jobs only, and turned into the history as prose.
The decision
- The ID is the class key, printed:
<source>or<source>/<alert_class>. The emitter declaresalert_classwhensourcealone spans more than one problem (automation_run_failure/<jobId>). Smith derives it only for posts it does not author (sentry/<Short ID>). No registry: a new class gets an ID by having asource. - One compose seam per runtime prints it.
notify()in each runtime ends a marked alert withAlert ID: <id> · ALR-<8 hex>, and the marker payload carriesalert_id. TheALR-alias is FNV-1a 32-bit of the id, identical in both runtimes and pinned by the same test vectors; it exists because Slack's tokenising of underscores could not be verified from the box. Hand-built CI cards add the same line. Until they do, Smith's router puts it on the first line of its thread reply. - Every fix carries it. Verdicts print the ID. Fix PR bodies end with
Alert-ID: <id>, so a human's fix counts as an attempt. - Fix history is derived, never kept. At routing time Smith reads the PRs for the ID from the last 30 days, merged or closed, from every author: those on its branch prefix, or carrying its trailer. The bare prefix counts only for a non-coarse, sub-class-free ID; otherwise only the trailer counts. It also reads the last three Smith verdicts for the source from the ledger and the dedup window's repeat count. A fix merged before the current firing, outside the owning repo's deploy grace, did not hold. Attempts made before the ID was printed count.
- A recurrence after a fix is re-investigated, not deduped. When the source has a fix that merged before this firing, the 48h duplicate skip is bypassed: the alert goes to the decider marked "RECURRED after #N", and one line is threaded under the earlier verdict.
- The ladder. No failed attempt: as today. One: the verdict opens with why fix #N did not hold, with evidence, and the fix widens to the class. Two or more: no patch. The decider writes a design doc PR (
docs/planning/alerts/<id>.md) naming every attempt and the bigger fix, and any human choice goes to the decisions page. "A different cause" may be argued with evidence. It does not reset the count. Both fix engines follow this. - Rejected: a new store for attempts (two records of one fact, ADR-0020); failure-notes.md as the store; wisdom.json (its per-domain cap evicts); a hashed opaque ID alone (unreadable, and not the key everything already joins on).
Consequences
- Pasting the footer into Slack or GitHub search finds every past firing, verdict and fix PR for the class.
- On day one
identity_contact_match_refusedsits at the top rung by design. Coarse classes (sentry,automation_run_failure,cloudwatch_alarm,channel-alert) start at zero until their emitters declare a sub-class. - App/CI occurrences reach the ledger through the daily reconcile, so counts between fixes are as of yesterday. The "did not hold" verdict depends only on the current firing.
- A failed GitHub read renders "history unavailable", never "no prior attempts". An empty result and a broken instrument must not print the same thing.
- The 48h dedup bypass costs one decider run per post-fix recurrence. That is the point: those are the runs that were being skipped.
- Not proven end to end until a real re-firing of
identity_contact_match_refusedcarries the footer, its decider block lists #20283 as not held, and its verdict opens with why.