A design proposal. Nothing here is built or decided — this page is the review surface.
src/app/api/voice/call-ended/route.ts:536-551
hands the transcript to handleVendorCallOutcome, which drops it — only the ElevenLabs
conversation id survives on transcriptRef. ElevenLabs retention is therefore our de-facto
retention policy, and the whole loop depends on a live vendor fetch. Fix: in
postcall-extraction.ts, write the transcript turns to a sibling DDB item
PK=PROP#{propertyId} / SK=VENDORCALLTRANSCRIPT#{sessionId} — a separate item, so the
session row stays small. Audio stays remote via fetchConversationAudio.
vendor-outbound.ts is not in
SPECIALIST_AGENTS, so sync-specialist-prompts.ts never pushes it.
simulate-battery.ts and real dials exercise the deployed agent, so a merged prompt
PR changes nothing live and the battery cannot verify it. Either add vendor to the sync list, or make
the manual push an explicit scripted step in the PR checklist.
VendorCallSessionCopy the WorkOrder.rating precedent verbatim (types.ts:2469-2476) — numeric/enum
verdict, verbatim free text kept separate, timestamp, actor.
| Field | Type | Notes |
|---|---|---|
callFeedback | 'good' | 'bad' | null | null = reviewed and explicitly dismissed; absent = not reviewed. Binary, never 1–5. |
callFeedbackNote | string | null | Prompt: "What should she have done instead?" Required on bad, optional on good. |
callFeedbackTurnRef | number | null | Index of the transcript turn the reviewer anchored to. Optional but high value. |
callFeedbackAt | ISO string | |
callFeedbackBy | string | Person/User id — same precedent as reviewedBy (types.ts:296). |
callFeedbackScenarioRef | string | null | Slug of the eval scenario minted from this call. |
All six go in the NewVendorCallSession Omit list, or they become required at creation.
ci.yml:211-230 — touching types.ts makes vitest --changed select nearly
the whole suite (35-min timeout), or blow AFFECTED_CAP and skip PR-lane testing
entirely. So: land the type + repo-interface change as its own tiny PR, then build the route and UI
on top of it. Two PRs, not one.
POST /api/vendor-outreach/[id]/feedback — a near-copy of
src/app/api/vendor-outreach/[id]/decision/route.ts: zod .strict() body
{ rating: 'good'|'bad'|null, note?: string, turnRef?: number },
requireRole(request,'conversations','edit'), getUserPropertyScope 403,
withErrorHandling. Differences: gate on a terminal session status (not
awaiting_approval), and write via patchVendorCallSession — no optimistic-lock path
needed. Re-rating is allowed and overwrites (last human wins); the raw prior value is not preserved, by choice.
Resolved calls are deliberately invisible today (QUEUE_VISIBLE_SESSION_STATUSES =
awaiting_approval | approved | cancelled, review-rows.ts:18-22) and no component
renders a VendorCallSession. So capture needs a host built, not extended.
Build a Recent calls tab in the existing review/HITL surface
(src/app/(workspace)/(operations)/review/page.tsx), one call per screen:
1 = good, 2 = bad, then a single free-text box, Enter advances. "Call 12 of 40" counter. Two-stage: the verdict commits on its own; the note never blocks it.Copy stays plain English throughout — no "rubric", "judge", "criterion", "eval". The button says Save; the box says What should she have done instead?
Not in MVP: dropdown taxonomy, star ratings, dashboards, voice memos.
A bad rating with a note mints a regression scenario immediately: an
evals/datasets/vendor-calling.yaml case named after the call
(vc-2026-07-22-hvac-no-eta), body = the reconstructed vendor situation from the transcript,
assertion = the note turned into an llm-rubric line plus any deterministic helper that fits
(assertBriefTurn, assertPlainLanguage in
evals/lib/assert-vendor-calling.ts). This is cheap, needs no judgment, and satisfies the standing
rule that every fix ships with a regression test.
A distillation run that proposes no prompt change has still produced value.
bad ratings accumulate. Not per-call. At Camellia's volume weekly would mostly produce empty runs; empty runs are normal, not failures, but monthly matches the "100 fresh traces per 2–4 week cycle" cadence better.entityType-createdAt-index scan, ENTITY_VENDOR_CALL_SESSION) + their transcripts + the current PROMPT + the current accumulated rule list + the standing off-limits guidance (no technical language in vendor-facing copy, no autonomous spend/send commitments, no ungating).Output per mode — one PR, one concern:
{ when: <scenario>, then: <behavior>, evidence: [sessionIds] }, rendered into the prompt with a strength tier — ALWAYS: / PREFER: / AVOID: — calibrated to how strongly the human actually spoke. Offhand preference = PREFER, never ALWAYS.evals/promptfooconfig-vendor-calling.yaml via run-all.sh), including the originating scenarios.| Guardrail | Rule |
|---|---|
| Bounded | Max 15 rules × 280 chars. At cap, retire oldest (FIFO) — the overlay rides in every call. |
| Amend, don't append | A recalibration replaces the superseded rule atomically, so two contradictory ALWAYS: lines never co-exist. Contradictions reconcile into a range, never last-wins. |
| Never-wipe floor | A distill that would empty a populated rule set is rejected as a model error. |
| Pareto guard | A rule that fixes 3 new scenarios and breaks 2 old ones is visibly rejected in the PR body, not merged. vendor-calling has no entry in evals/domain-baselines.json, so it runs at the strict default floor — a new failing case reds the domain immediately, which is the behavior we want. |
| Prompt-var drift | Any new {{var}} needs an emitter in vendor-call-context.ts in the same PR, or vendor-prompt-vars-drift.test.ts fails. |
| Never auto-merge | These PRs change customer-facing agent behavior — the same hold-for-Fede category as ADR PRs. CI green is necessary, not sufficient. |
| Held-out set | 20% of rated calls are reserved and never used to generate rules. It is the only way to tell "the rule generalizes" from "we overfit to nine calls". |
The job posts to #updates-fede: the failure mode, the counted evidence, the proposed rule with its scope, the eval delta. Fede's ✅ is what opens the PR — approval and PR creation are one gesture, not two. Silence = no PR. Every landed change gets a one-line "what changed and why" post.
Silent learning is the failure mode.
Now
Binary verdict + note, in the review UI. Metric: calls reviewed per sitting, and % of bad ratings carrying a note (target ≥60%).
After ~50 labels
Clara scores every vendor call against a small written rubric before a human sees it — did she confirm scope, get a firm ETA, avoid committing to a price, capture a callback number — each verdict pass | fail | unknown with a required rationale quoting the transcript line. The reviewer then agrees or disagrees per line instead of composing from scratch. unknown is a legal verdict; an eval that can't tell must never count as a pass.
Before any dimension runs unsupervised
Log Fede's verdict beside the judge's and compute Cohen's κ, not raw agreement. κ ≥ 0.6 on a dimension is the bar for letting that dimension run unsupervised. The existing Clara conversation-eval loop sits at κ = 0.454 with diagnosed over-strictness — expect the same failure and the same fix (few-shot anchors drawn from over-flagged passes).
Only once the judge is calibrated
PM capture is a single thumbs on the outcome notification they already receive (notify-pm-outcome.ts emails/SMSes the property PM with a /work-orders/{id} link — a signed thumbs link there is the lowest-friction point in the system). PM feedback is routing signal into the review queue, not a direct input to distillation: a PM thumbs-down surfaces a call for Fede, it does not vote on a prompt rule. Per-property gate on the whole thing, matching the outreach-cadence precedent.
good calls too, so we can measure the true-negative
rate and not only failures.
Feedback entity / new tablepatchVendorCallSession, done.| # | Pattern | Source | Verification |
|---|---|---|---|
| 1 | Thumbs + free-text comment stored on the conversation record, separate from the machine's own verdict; unknown is a legal judge outcome and every verdict carries a rationale. Human signal is never overwritten by the judge. |
ElevenLabs (metadata.feedback → overall_score like/dislike + nullable comment; success-evaluation success|failure|unknown + rationale) |
Confirmed — read directly from the API schema and Analysis docs. Our claim is a correct subset of the real shape. |
| 2 | "Create test from this conversation" — a failing real call converts into a prefilled regression scenario; results reported as pass-rate bands (green 100 / amber ≥80 / red <80) with failure bucketing, not a boolean. | ElevenLabs Agent Testing (repeat_count 2–20, grouped-by-failure-reason results) |
Confirmed — exact UI strings and band thresholds verified in docs. |
| 3 | Proposal-with-diff in pending state, approved by a human in Slack or dashboard; the coach never applies changes automatically. This is our approval flow almost verbatim. |
ElevenLabs Coaching | Confirmed — "The coach does not apply changes automatically", pending → Approve/Reject, Slack review, diff over prompt/memories/procedures, all verbatim. |
| 4 | Validate before asking: test each proposal against the originating conversation and the full golden set, and put issue → evidence → diff → validation results in the review surface so approval is one judgment, not an investigation. Human approval required before production. | Decagon Duet Autopilot | Confirmed — the load-bearing sentences quoted from Decagon's own post. "Overnight" is positioning language, not a documented schedule, so our monthly cadence is our own call. |
| 5 | Every rule carries a scope condition the reviewer argues with, auto-suggested by the model and edited by the human, kept narrow — one scenario per rule, never a catch-all. | Ada Coaching ("When replying to" field; best-practice: "as specific as possible… only a singular scenario at a time") | Confirmed — verified in Ada docs. Ada ships this with no approval gate and documents no conflict resolution or volume cap; we add all three deliberately. |
call_successful side by side with nothing reconciling them. We are doing this because our own
conversation-eval ADR already proved the metric bites (κ = 0.454, below the 0.6 bar), not because anyone
else does it.
These are open, not settled. The design commits to a position on each, but every one is a live call for Fede.
vendor-outbound.ts to SPECIALIST_AGENTS, or keep the push manual as a scripted PR-checklist step. Until one is chosen, the improvement half of the loop cannot be verified.bad on-demand trigger? Weekly at Camellia's volume would mostly produce empty runs.bad ratings carrying a note. These are proposed values, not measured ones.
Status: Proposed — awaiting Fede's review. Nothing described here is built, merged, or
decided. This page is faithful to the source design note and adds no numbers, dates, or commitments beyond
it. Source: vendor-call-feedback-loop.md.