0056 — A script that fell back to AI becomes a script-fix PR, through Smith's existing alert decider

• Status: Accepted 2026-10-14. Built in agent-smith #1089. ⚠️ THE NUMBER 0056 IS SHARED BY TWO ACCEPTED DECISIONS — a bare "ADR-0056" does not identify one. Cite this record as ADR-0056 script-fix-pr; the other is ADR-0056 agent-smith-channel (0056-what-agent-smith-channel-carries.md). Not renumbered, on purpose: ADR-0023 decision 6 and ADR-0025 Open 3. The one record of this, with the citation counts, is NUMBER-REGISTER.md; relay/tests/adr-register.test.ts fails if it goes stale.
• Date: 2026-10-14
• Deciders: Gera asked for it (2026-10-14): when a script breaks and the AI fallback takes over, the #alerts post should get Agent Smith to fix the script. "We should already have a lot of that logic where, for example, on alerts, Smith picks it up." Fede owns the AppFolio robot's login flow and can object.
• Systems: the AppFolio robot (appfolio-browser-agent, lib/aiFallbackAlert.ts), PropFlow's Sentry project and its "new high-priority issue, production" rule to Slack #alerts, Smith's Slack socket receiver (smith-slack-socket) and its alert decider. Clara (propflowai) is next on the same pattern.

The problem

The robot now runs script first, AI as the fallback (appfolio-browser-agent #515, #520). When the script does not recognise a page, the AI takes over, and reportAiFallback raises an AiFallbackError in Sentry. The event's fingerprint is (flow, step, page). So each new cause opens one Sentry issue and gets one #alerts post, and later repeats only add to that issue's event count. The event carries the step, why the script did not match, and a Browserbase session id. That session's log holds the page snapshot (DOM.getDocument) for about 7 days.

A post nobody acts on is just a log line. The script gap stays open, and the AI keeps carrying a step the script should own.

What already existed (most of it)

This ADR adds a new kind of alert to an existing path. It does not build a new path.

• Smith already watches Sentry posts in #alerts. agent_smith/sentry_alert.py lets a Sentry issue post in when two things are true: the author is on the allowlist (SENTRY_BOT_UIDS) and the post parses as an issue. slack_socket.py checks this before it drops bot posts (_route_sentry_alert, then _route_alert). The admission is live: the socket log has 335 slack_socket sentry alert lines, the latest on 2026-10-14.
• _route_alert runs the whole alert protocol. In order: 👀 ack, ignore/mute gate, dedup on the Sentry Short ID, episode and class-fold gates, then the alert decider. The decider is a bounded Smith run (alert_remediation_prompt.build_alert_decider_prompt). It investigates, posts a verdict under the alert, and on a code fix opens a PR off origin/main and drives it to mergeable. Merging follows the §5 tiered gate: Smith may merge only low-risk work, and anything else needs a human ✅ on the alert.
• The robot shares propflowai's Sentry project (lib/tenancyAlert.ts), so its posts come from the same Sentry app and enter through the same door.
• The note "no agent route mints a Smith kickoff" does not apply here. It is about operator kickoffs from an agent's Slack post or the CLI. A Sentry issue post reaches the decider through its own admission path, and that path already works.

What was missing: the decider had no idea a script gap existed. Its §4 named only agent-smith and propflowai as repos to fix in, so a robot fix would have been opened in the wrong repo. Nothing told it that the why and the Browserbase session id sit in the Sentry event rather than in the Slack post, or that it must never log in to see the page. And every Sentry issue shared one fix-branch prefix (smith-fix/sentry-sentry), so its resume-search could pick up an unrelated open fix PR.

The decision

1. Trigger: a new cause, not a count. The first event of a new (flow, step, page) fingerprint opens a new Sentry issue, which makes one post and one decider run. A single AI takeover already proves the script is missing a page, so no threshold is needed to justify the fix. A repeat of the same cause lands on the same issue, and the existing Short-ID dedup sends it to the run that already owns it. If a count threshold is ever wanted, for example "only after 3 occurrences", it is a Sentry alert-rule condition (event frequency), not code. Regression after a fix is the rule's reappeared_event trigger. That is Sentry-side configuration and outside this ADR.
2. Recognition: by exception type. A Sentry issue titled AiFallbackError is a script gap. The project label cannot tell the robot from propflowai because they share a project, so it is not used. The decider picks the owning repo from the event's stack: lib/aiFallbackAlert.ts means the robot, and a propflowai file means Clara.
3. What the task carries. The decider gets the Sentry issue link and Short ID, plus a playbook appended to the alert detail. The playbook says to:
   • pull the latest event (/organizations/<org>/issues/<id>/events/latest/) for its tags (flow, step, where, browserbaseSessionId) and its alert context, which holds the why;
   • name the owning repo and its gate;
   • pull the page from GET api.browserbase.com/v1/sessions/<id>/logs;
   • turn the snapshot into a scrubbed fixture (robot: tests/fixtures/login-screens/) and a test that fails before the fix.
4. Dedup. Within one cause, dedup is by Sentry issue (the Short ID, already in place). Across causes, AI-fallback alerts get their own kind (ai-fallback), so their fix branches are smith-fix/ai-fallback-sentry-…. They never resume a non-script Sentry fix PR. An open AI-fallback PR is resumed only when it covers the same flow and step. A different step gets its own PR.
5. Who merges: a human. The script fix is a normal PR. It goes through the repo's claude-bot review to a clean 🟢. By the decider's §5, the robot's scripts are HIGH risk because they drive a customer's PMS, so Smith never merges them itself. A human merges with a ✅ on the alert. A robot PR that needs live proof keeps its [HOLD: live proof], and a human does that proof. (Amended 2026-10-31 by ADR-0078: the risk is now the reviewing seat's label, not Smith's §5 tier. The robot's scripts are on its HIGH list, so the PR still goes to a person, paged by the PR drive.)
6. Guardrails.
   • Never log in to a customer's AppFolio (or any customer system) to see the page, and never start a new browser session to reproduce it. The saved Browserbase snapshot is the only source.
   • No snapshot means no guess. If the snapshot has expired (more than about 7 days) or there is no session id, the verdict is sync eng (a decision raised to a person), not a login and not a guess.
   • Fixtures are scrubbed. No credentials, MFA codes or tenant PII go into them.

Consequences

• One AI takeover now produces one PR in the right repo, carrying a fixture and a test, ready for a human ✅. No new poller, no new channel and no new route was added. The work is a recognition rule and a playbook inside the existing Sentry admission and decider.
• Not yet proven end to end. No real AiFallbackError post had reached #alerts when this was written, so the test for it is built from a real Sentry capture with the title swapped. The first real one is the proof. Check that it got a 👀, a verdict in the thread, and a PR in appfolio-browser-agent on a smith-fix/ai-fallback-sentry- branch.
• Open risk: does the post fire at all? It depends on Sentry treating a new AiFallbackError issue as high priority, because the #alerts rule is "new high-priority issue". The robot raises it as an exception, which Sentry usually rates high, but this has not been measured. If the first fallback never posts, the fix is in the Sentry rule, not in Smith.
• Credentials the decider reads: Sentry from propflowai's .env.local (as before), and Browserbase from the robot's .env.local (BROWSERBASE_API_KEY, present on the mini). Both are read in place and never copied.
• Clara uses the same type name (AiFallbackError) and the same fingerprint scheme, so it is covered with no change here.