0064 — Inbound Audio Voice Note Transcription via ElevenLabs Scribe

• Status: Proposed (held for review — not merged)
• Date: 2026-06-19
• Deciders: Fede
• PR: fede/inbound-voice-note-dropped

Context

A PM replied to a turnover recap SMS with an audio voice note sent as an MMS (MediaContentType: audio/amr, empty Body). The voice note was silently dropped — no error surfaced, no retry, no record of the audio content.

Evidence:

• Conversation: conv_voice_b1e592f5-c0ce-4ef5-bece-408a7bea5f03
• Outbound SMS: SM922daa4bf84cf259c6075fc526b7b6f1
• Inbound MMS: MM561bd9876b63adc29e6876317f568711 (MediaContentType audio/amr)

Root Cause

Two lines in src/app/api/twilio/webhook/route.ts conspired to drop the message:

1. Line ~224 — the media processing loop skipped non-image media with if (!mediaContentType.startsWith('image/')) continue;. An audio/amr voice note never entered the images array.

2. Line ~244 — after the loop, the handler returned HTTP 400 when the message body was empty and the images array was empty:

   if (!messageText && images.length === 0) {
     return NextResponse.json({ error: 'Missing Body and no media' }, { status: 400 });
   }
   

Twilio does not retry 400 responses (unlike 5xx). The voice note was permanently lost with no user-visible signal and no Sentry alert.

Existing Capability

ElevenLabs Scribe (/v1/speech-to-text, scribe_v1 model) is already integrated in this codebase:

• agents/clara/lib/voice/post-transfer.ts — transcribePostTransferRecording downloads dual-channel WAV recordings from Twilio and transcribes them via an unexported scribeMono helper that POSTs to https://api.elevenlabs.io/v1/speech-to-text.
• src/app/api/admin/agent-smith/run/route.ts — audio_transcribe action (display helper, references the same capability).

The Twilio MMS webhook just never routed inbound audio into this path.

Decision

Transcribe inbound audio MMS via ElevenLabs Scribe, then route the transcript through the normal message pipeline.

New module: src/lib/integrations/voice/scribe-transcribe.ts

Extracts the Scribe call pattern from post-transfer.ts into a reusable, exported function:

export async function transcribeAudioBuffer(
  audioBuffer: Buffer,
  filename: string,    // e.g. "voice-note-MMxxx.amr" — hints codec to Scribe
  contentType: string, // e.g. "audio/amr"
  apiKey: string,
): Promise<ScribeResult>

No side effects, no data writes. Returns { text: string }.

Webhook changes: src/app/api/twilio/webhook/route.ts

The image/-only branch becomes three branches:

• image/* — existing behavior (pass to Claude vision).
• audio/* — download via fetchTwilioMedia, transcribe via transcribeAudioBuffer, push transcript string to audioTranscripts[]. On Scribe failure: log the error, push a placeholder '(voice note — could not transcribe)' so the message still routes.
• All other MIME types — log and skip (existing behavior).

After the loop, messageText is assembled with priority:

body > audioText > '(photo attached)' > ''

If messageText is non-empty, formFields.Body is patched to the computed value before adapter.parseInbound is called, so the MessageEnvelope.body carries the transcript through the SQS/Lambda/processEnvelope pipeline.

Error handling: Scribe failures push a placeholder and continue — the message always routes and Twilio always gets a 200. A 400 would be unretried by Twilio and the voice note would be permanently lost; the fix avoids that failure mode even on partial outages.

Consequences

Positive:

• PM voice notes are transcribed and processed by Clara, completing the PM feedback loop for MMS-based interactions.
• Scribe failures degrade gracefully: the PM at least knows their voice note arrived (placeholder text), and ops sees the error in CloudWatch + Sentry.
• No new external dependencies — reuses the existing ElevenLabs API key and Scribe integration.

Neutral:

• ELEVENLABS_API_KEY must be set in production. If absent, the handler logs a warning and pushes '(voice note — transcription unavailable)'. The message still routes.
• Adds one Scribe HTTP call per audio attachment (parallel with any image fetches). Scribe latency is typically 1–3s for short voice notes; still well inside Twilio's 15s webhook deadline.

Negative / Risks:

• Audio transcription cost: Scribe charges per minute of audio. PM voice notes are expected to be short (under 30s typically). No volume guard is in place. If PMs start sending long audio, add a size check before calling Scribe.

Alternatives Considered

1. Acknowledge audio without transcribing — return 200 + placeholder text. Rejected: the PM gets no useful feedback, and the action remains unactioned.

2. Store raw audio URL and flag for async transcription — Twilio media URLs expire in 4h without re-authentication. Deferred transcription is complex and would require a separate job queue. Not worth the complexity for PM voice notes.

3. Use Twilio's built-in transcription — Twilio Transcriptions uses a different API and produces lower-quality output. Scribe is already battle-tested in this codebase on the post-transfer recording path.