0115 — One eval catalog: register every eval, gate it from one place

• Status: Accepted
• Date: 2026-07-27
• Deciders: Gera (+ the 7-red-domain investigation, docs/planning/clara-eval-sweep-7-red-domains.md)

Context

PropFlow has a lot of evals and no idea how many. Measured on origin/main today, not estimated:

Two independent audits of the above (2026-07-27) corrected the framing twice, which is itself the argument for a registry:

• There are SIX runners, not two. Besides run-all.sh and nightly.yml, three dedicated path-filtered per-PR workflows run individual configs (renewal-evals.yml, renewal-evals-sms.yml, renewal-evals-scoping.yml), plus the maintenance Temporal lane. Modelling only two is what hid the WORST threshold conflict in the repo: renewal-scoping is gated at 95.0 by its own per-PR workflow and at 60.0 by nightly, against a recorded measurement of 80.0% — a 35-point spread, so a PR touching its paths is gated 15 points above what the suite demonstrably scores.
• The sweep runs 43 domains, not 38 — run-all.sh also runs five non-promptfoo domains (tools, spine, channels, routing, maintenance-judge).
• There is a fourth threshold store: check-eval-rate.mjs carries DEFAULT_THRESHOLD = 95.0, latent today because every caller passes a value.
• Some gates cannot fail at all. voice-comprehensive.yml passes --soft unconditionally and --soft exits 0 before the gating check, so a 118-scenario voice harness on a PR + nightly cadence has never been able to red CI. Its own workflow header says flipping --soft off "lands a known-green gate set"; that has not happened. An eval that runs but cannot fail is indistinguishable from a passing one without a registry that says so.
• Zero promptfoo configs are dead. Both "no runner" configs turned out to be MISWIRED — shipped alongside the production fix they guard and never added to a runner — not abandoned. The consolidation work is wiring and threshold reconciliation, not deletion. The one genuinely superseded surface, the eval:voice / eval:pull / eval:batch transcript trio (absent input directory, divergent judge models, retired mono-agent env), is DELETED in the PR that introduces this ADR — scripts and npm aliases together — per rule 6. An earlier revision of that PR registered it as state: 'retired' and deferred the deletion, which this ADR's own rule forbids.

Nothing in the repo knows those rows describe one population. The consequences are not hypothetical — they are the four findings of the 2026-07-27 investigation into seven "red" Clara domains:

1. Two gates disagree about the same domain — for THIRTY domains, not four. leasing-concerns is GREEN at a 78.0 floor in nightly.yml and RED at an implicit 100% bar in the sweep, because domain-baselines.json has no entry and the sweep's documented default for an unlisted domain is STRICT. The catalog's inForce audit puts the real count at 30: four are number-vs-number (leasing-response 81/78, tenant-resident-services-gauntlet 84/74, turnover-walk-editing-gauntlet 79/86, renewal-scoping 95/60) and 26 more are measured-nightly-floor vs implicit-strict-sweep. An earlier draft of this ADR said four, having counted only the first kind.
2. A manifest promises a tolerance the gate cannot deliver. evals/tracked-gaps.json declares three tolerated gaps under leasing-concerns and its own _meta says the floors in domain-baselines.json tolerate them. That file has no leasing-concerns key. The sweep prints those gaps as "tolerated" in the same summary where the domain is red because of them.
3. A model migration silently orphaned an entire suite. Every provider entry is a hardcoded model string (66 of them: 41 claude-sonnet-4-6, 23 claude-haiku-4-5-20251001, 2 undated claude-haiku-4-5). When #4688 flipped AGENT_MODEL to claude-opus-5, every text/SMS eval kept grading Sonnet. Nothing failed, nothing warned; the leasing suite simply stopped describing production. Two config headers went on asserting "Provider matches production EXACTLY" for another two days (three more asserted the same thing in different words).
4. A floor set from one sample. leasing-on-behalf carries a 95% Sonnet floor over 17 rows, recorded with no measured baseline and no guard band. Observed 13/17 and 14/17 on byte-identical inputs — one flipped row moves it while the floor permits zero misses. It cannot be cleared by prompt work.

There is also no way to answer "what evals do we have, and which are green?" /admin/dev/evals has eight hand-maintained tabs; its tab list is mirrored between page.tsx and EvalsShell.tsx by hand. Live-bench harnesses (Morpheus / Maestro, the maintenance Temporal lane, voice-harness, pipeline-lab) are not represented at all.

This is the same class of problem the Metrics Platform (METRIC_CATALOG) and the Tools Platform (src/lib/tools/) already solved: one TypeScript-enforced catalog, one UI, drift guards that fail CI when an entry is added without being wired. Evals never got that treatment.

Decision

Every eval in the repo is registered in one catalog, src/lib/evals/catalog.ts, and every gate, schedule, and UI surface derives from it. An eval that is not in the catalog does not exist; a catalog entry that is not wired fails CI.

What the catalog does NOT do

It does not merge the runners. Three genuinely different classes exist and keeping them separate is deliberate:

A bench mutex has no business near a $0.15 prompt eval. The catalog unifies the contract — pass bar, result shape, model resolution, cadence, ownership — not the execution.

The one contract

Each entry declares:

• key — stable slug, the join key across gate / workflow / UI.
• class — prompt | deterministic | live-bench.
• runner — the exact entrypoint.
• surface — what production behavior it grades (e.g. clara.leasing.sms).
• tier — the model tier the surface ships on (AGENT_MODEL | FAST_MODEL | elevenlabs | pinned). Not a model string.
• gate — { measured, guardBand } or strict, the single source of the pass bar, per-provider where the domain scores per-provider.
• cadence — pr | nightly | manual.
• isolation — none | bench-lock | queue-override | dedicated-property.
• state — live | manual | quarantined | retired, mirroring the metric catalog's lifecycle axes.
• advisory — set, with a reason, when the eval RUNS but structurally CANNOT red its lane. The most dangerous state an eval can be in and the one most invisible without being written down.
• thresholdConflict — the source LIST (not a pair) when different runners gate the same eval at different floors.
• owner — who fixes it when it reds.

The rules that follow

1. One threshold. nightly.yml's 41 inline numbers and domain-baselines.json's 5 entries collapse into gate. Both gates read the catalog. Two files can no longer disagree.
2. Guard bands are mandatory for a rate gate. domain-baselines.json's own _meta.howToSet already says minPassRate = measured − guardBand; the catalog enforces it in the type, so a floor 0.26 points under a measured rate is not expressible.
3. A policy floor is distinguishable from a measured one. EvalPolicyBar requires a rationale. Conflating the two produced the leasing-on-behalf defect: a floor of 95 recorded with no measured baseline and no guard band, under a note reading "Sonnet meets the strict bar", which every later reader took for a measurement. Sonnet measures 76.5–82.4%.
4. Tiers resolve, models don't get hardcoded. An eval says which surface it grades; the provider comes from the tier. A future AGENT_MODEL flip re-points its evals or fails loudly — it cannot silently orphan them again.
5. Coverage is total. Every promptfooconfig-*.yaml on disk has an entry; every entry points at a file that exists; no threshold may be written inline in a workflow. Drift-guarded.
6. state: 'retired' is a deletion order, not a parking space. Consistent with the repo's ONE SOURCE OF TRUTH rule, an eval that is dead gets deleted with its config, dataset, bridge and thresholds in the same PR.

The Overview surface

/admin/dev/evals gains Overview as its first tab — the honest high-level answer to "what do we have and what's failing", modeled on /admin/dev/status's card grid and reusing its StatusLevel vocabulary (healthy / degraded / down / unknown). One card per catalog entry, grouped by class, showing last result, pass rate vs. its gate, cadence, and staleness. unknown is a first-class state: an eval that has never run must read as "no signal", never as green — the same discipline the sweep's own workflow header states ("a skip is NOT a pass, and must never look like one").

Entity classification (ADR-0027)

Consequences

Easier. One place to answer "what evals exist". One place to change a pass bar. A model migration can be audited against the catalog instead of by grepping 66 hardcoded strings. New evals get a home and a gate by construction. The Overview tab makes a red domain visible to someone who is not running the sweep.

Harder. Adding an eval becomes a two-file change (config + catalog entry) instead of one, and the drift guard will block the second file being forgotten — deliberately the same friction the metric catalog imposes. Migrating nightly.yml to generated thresholds touches a workflow that is currently hand-tuned in 41 places, and that migration must be done without changing any effective threshold in the same PR that moves them.

Follow-up implied. (a) The two-gate collapse, which is the change that actually fixes four of the seven reds. (b) A promptfoo provider that can call Claude 5 — the pinned 0.121.19 sends temperature unconditionally and Claude 5 400s on it, so tier: 'AGENT_MODEL' cannot be executed until that is solved, only declared. (c) Re-measuring leasing's baseline before it can be given an honest floor. (d) Deleting what the catalog exposes as dead.

Explicitly not committed to. Merging runners, a bespoke eval-execution engine, or moving the live-bench harnesses off their current isolation mechanisms.

Alternatives considered

Leave it; fix the seven reds individually. Rejected: it is the fourth time this class of drift has been repaired by hand, and the repair does not survive the next surface being added. The investigation this ADR comes from found the same root cause behind four separate "failures".

One mega-runner for all evals. Rejected: it would put a bench mutex and a Temporal queue override in the path of a stateless prompt eval, and would make the cheap per-PR lane as fragile as the most expensive harness.

Generate the catalog from disk (glob the configs, infer the rest). Rejected: the fields that matter — surface, tier, gate, owner — are judgments, not facts recoverable from a YAML file. Inference is how tracked-gaps.json and domain-baselines.json drifted apart in the first place.

Put the thresholds in DynamoDB so they can be tuned without a deploy. Rejected for now: a pass bar is a reviewable decision and belongs in the diff. Revisit only if threshold churn becomes the bottleneck.