0038 — Meta glasses as a client-side capture UX for the existing turnover intake

• Status: Proposed
• Date: 2026-05-27
• Updated: 2026-05-28 — hardware & SDK research pass (see Update — 2026-05-28); 2026-05-30 — V2 native DAT app built & proven end-to-end on real hardware (see Update — 2026-05-30)
• Deciders: Fede
• Related: ADR-0034 (turnover orchestration; owns ConditionReport / ConditionProjection and Slice 3 Clara-native intake), ADR-0026 (PII at the boundary), Trello epic BoW61MAW

Context

This is a thin client-side UX layer, not an architectural change. The entire backend pipeline this depends on already exists. Adopting Ray-Ban Meta smart glasses adds one input device and one PM-side iOS Shortcut on top of the unchanged Slice 3 condition-report intake.

What's already in the codebase (no change proposed by this ADR)

A PM today can already do the full walk by voice call + SMS-with-photos: dial the turnover-intake line, narrate, text photos in, watch them attach to the draft ConditionReport. The glasses change only how the PM performs that exact same flow.

What this ADR proposes (the entire delta)

1. One iCloud Shortcut, hosted at iCloud Shortcuts (versioned by replacing the link, not by repo). Triggered by an iOS Personal Automation on Bluetooth pair with Ray-Ban Meta. Action: poll the iOS Photos library filtered on Ray-Ban Meta EXIF metadata, send each new photo as an MMS to the existing turnover-intake inbound number. That is the whole client.
2. One screen-recording / onboarding doc showing the ~60-second setup (Shortcut add + Personal Automation + Photos/Messages permission grants).
3. No new code in this repo. No new entity. No new agent. No new tool. No new endpoint. No schema migration. The MMS lands on the same webhook that already accrues photos onto the draft ConditionReport.

The user-stated UX, mapped to the existing pipeline:

Hard constraints from the user

• No "Hey Meta" per photo. Temple button is the only per-photo action.
• No third-party app on the PM's personal phone. iOS-native features (Shortcuts, Personal Automations) are acceptable.
• One-time setup ≤ ~60 seconds per PM.
• 2-day delivery budget.

Research summary — Meta wearables developer surface (mid-2026)

• Meta Wearables Device Access Toolkit (DAT) is in public developer preview since 2025-12-04. GA "later in 2026." Native iOS Swift + Android Kotlin SDKs (facebook/meta-wearables-dat-ios, facebook/meta-wearables-dat-android). Distribution capped at ≤100 testers per release channel until GA.
• DAT exposes programmatic photo capture (session.capturePhoto()), video stream (session.videoStream.collect, ≤720p @ 30fps over BLE — the ≤720p/30fps figure is unverified as of 2026-05-28; not stated in Meta's public FAQ, see Update), mic as standard Bluetooth headset (no raw multi-mic PCM).
• No custom "Hey Meta, ask <app>…" voice intent. Meta explicitly excluded Meta AI from DAT. (Unverified as of 2026-05-28 — could not confirm in Meta's public FAQ; treat as a claim to re-check against primary docs.)
• DAT's session.photoDataPublisher is a Combine publisher; whether user-pressed temple-button photos emit on it while a session is active is undocumented. Resolvable in ~2h of test code on real hardware.

Critical finding — folded-only photo sync

From the official Meta help docs, stated consistently:

"Auto-import will occur when your glasses are switched on and are in the charging case or their hinges are closed, and your phone's Bluetooth and Wi-Fi are on."

Photos do not sync to phone while the PM is wearing the glasses. They buffer on the glasses (up to 500 photos / 25 30-second videos per charge) and only transfer over the glasses' temporary Wi-Fi when the hinges close. The V1 Shortcut therefore picks up photos in a batch when the PM folds the glasses, not mid-walk. ADR-0034 Slice 3 makes no real-time requirement; the PM-approval freeze runs after capture either way.

Decision

Adopt a two-tier rollout, both tiers under ADR-0034 with zero backend changes.

V1 (2-day delivery): iOS Shortcut → existing inbound MMS pipeline

• PM dials the existing turnover-intake voice agent through the glasses BT headset.
• Bluetooth-pair Personal Automation kicks off a Shortcut that polls iOS Photos for new Ray-Ban Meta captures and forwards each as an MMS to the same inbound number that owns the voice call.
• The existing webhook attaches the photo to the in-flight draft ConditionReport via the existing accrual path (src/lib/tools/turnovers.ts).
• Because of the folded-only sync, photos batch in at end of walk. Voice notes arrive in real time.

V2 (post-MVP, gated on Meta DAT preview admission): real-time photos via DAT BLE

• Native iOS Swift app using facebook/meta-wearables-dat-ios. Foregrounds a DeviceSession, subscribes to session.photoDataPublisher.listen { … }.
• If user-initiated temple-button photos emit on the publisher (Scenario A; ~2h to verify on hardware), V2 ships real-time photo upload to the same Slice 3 capture endpoint with the same write semantics.
• If they do not (Scenario B), V2 collapses; V1 remains. Backend contract unchanged either way.

Entity classification

This ADR proposes zero new export interfaces. The entire change is a client-side capture UX producing existing ADR-0034 entities. Per ADR-0027 the classification table is therefore omitted.

Consequences

Easier

• 2-day delivery is realistic because no PropFlow code changes. Day 1 is the Shortcut and onboarding doc; Day 2 is hardware testing + iterating on the EXIF filter.
• Existing Slice 3 invariants — immutable lease-anchored ConditionReport, S3 Object-Lock evidence bucket, pointer-keyed vision path, PM-approval freeze, eval ship-gate (evals/promptfooconfig-turnover-damage.yaml) — all hold without re-validation because nothing in this ADR touches them.
• V1 and V2 share the same backend contract; V2 is a pure transport swap.

Harder

• V1 photos are batch at end of walk, not real-time. ADR-0034 makes no real-time requirement. V2 closes the gap if DAT preview admits us.
• iOS Shortcut reliability under iOS background-execution limits depends on the PM's iOS version. Mitigated by Phase 0 experiments and by the loop running while a phone call is active (phone stays awake).
• Twilio MMS per-photo cost ~$0.02 USD per US MMS message (pricing source); ~$1 per 50-photo walk. Negligible at current volume.
• Per-PM onboarding adds ~60s of setup. Mitigated by a short screen-recording.
• Shortcut's EXIF filter depends on Meta tag values not officially documented; needs hardware verification (Phase 0).

Follow-up work

• Apply to Meta Wearables DAT developer preview (parallel track, no V1 dependency).
• Phase 0 experiments on real hardware to resolve the open questions below.
• V2 DAT iOS app: ~3 weeks of work, gated on DAT preview admission. Trello In1z0MTo.
• V2 stretch (continuous video stream + smart frame capture on verbal cue): Trello vqzTDWap. Out of scope.
• Display HUD overlays on Ray-Ban Meta Display via DAT + Web Apps SDK: out of scope until V2 lands and Display hardware is in inventory. Note (2026-05-28): the Web Apps SDK is not future — Meta opened it publicly on 2026-05-14 (13 days before this ADR), and it's explicitly Claude-Code-friendly. But Web Apps have no camera/photo access (native-DAT-only), so the in-lens web app can be a HUD surface, not a photo transport. See Update — 2026-05-28. The remaining blocker is hardware-in-inventory, not SDK availability.

Open questions to verify on real hardware

1. Exact EXIF camera-make / camera-model strings on Ray-Ban Meta captures — determines the Shortcut's Find Photos filter. ~5 min.
2. iOS Shortcut "Send Message" auto-send without per-message confirmation for MMS — if iOS requires confirmation per send, fall back to HTTP multipart POST to the existing turnover endpoint. ~15 min.
3. Personal Automation reliability with "Run Immediately" on Bluetooth connect — 10 pair/unpair cycles in mixed states. ~30 min.
4. Shortcut polling loop survival over a 30-min walk — phone in pocket, screen locked, voice call active. Measure termination latency. ~45 min.
5. End-to-end latency from "glasses folded" → photo in iOS Photos library — determines staleness of report finalization. ~10 min.
6. (V2 scoping) DAT session.photoDataPublisher emission on user-pressed temple-button — ~2h with DAT preview slot + minimal sample app.

Update — 2026-05-28: hardware & SDK research pass

A follow-up research pass (web verification against Meta's developer + consumer docs, prompted by field-testing the manual flow — a full turnover by voice call with stills texted in worked end to end) confirmed the load-bearing limitation but surfaced one stale framing, two missing hardware constraints, and two better no-app transport options. The V1/V2 decision above stands; this section amends the supporting detail and corrects a few claims.

Confirmed

• Folded-only / batch sync holds. Re-verified: captures buffer on the glasses' onboard storage and transfer to the phone only when the glasses are docked / hinges closed with BT + Wi-Fi on. No streaming transfer while worn. Real-time capture off the glasses exists only via the native DAT BLE path.
• DAT distribution limits hold. Still developer preview; ≤100 testers per release channel; no public App Store / open publishing yet ("in 2026", no firm date).

New constraint the original ADR missed — the Web Apps path has no camera access

On 2026-05-14 Meta opened the Ray-Ban Display glasses to third-party developers with two build paths. The two are not interchangeable for our purpose:

Camera and photo/video capture are native-DAT-exclusive. A Display Web App cannot reach the camera or the phone's photo library, so the tempting "host a TypeScript page in-lens that grabs the last capture and POSTs it" pattern is not buildable as a Web App — anything touching a photo needs the native companion (full app + preview admission + per-phone install). The Web App's real role here is a heads-up display surface (in-lens turnover checklist, "photo received ✓", which-unit status, driven by Neural Band / GPS), not a photo pipe. Recorded so the camera-less fact isn't rediscovered the hard way. Still hardware-gated.

New constraint — the microphone is single-owner; no video while on the Clara call

The glasses' mics are committed to the call audio while the PM is on the turnover line (glasses = BT headset). Per Meta's FAQ:

• Photos coexist with an active call — a still needs no mic. This is exactly the validated flow (narrate to Clara, tap the temple for stills).
• Video does not — it records audio through the same mics; Meta states capture is blocked while the mic is in use and that calls "interrupt recordings." A clip started mid-call will fail / drop / interfere with the call.
• Voice-command capture is also blocked mid-call ("Hey Meta, take a photo" isn't heard); the temple button still works for stills.
• The only live-video exception is "share your view," which streams POV inside a Meta video call (WhatsApp/Messenger/IG) — not a regular phone call, and the streaming path this ADR already rejected.

Net: stills via the temple button are the only capture that runs concurrently with the call — the medium V1 already uses. Video buys nothing (same folded-only sync, heavier to move, on-glasses clip-length cap).

Better no-app transport options for V1 (supersedes "Shortcut → MMS" as the only path)

All three need no native app and all three inherit folded-only batch (the photo must reach the iOS camera roll first). The ADR previously named only the Shortcut:

1. iOS Shortcut → MMS or HTTP POST (original V1). Mechanics below. Weakness: iOS background-execution reliability + a non-shareable automation (see Corrections).
2. Cloud-backup webhook (recommended for a real rollout). PM enables Google Photos / Dropbox / iCloud auto-backup of the camera roll (an OS-level daemon, far more reliable than a Shortcut). A server-side watcher we own (Google Photos Library API, or Dropbox webhooks + longpoll) pulls new Ray-Ban Meta captures filtered by EXIF and feeds the existing accrual path. No phone app, no per-phone automation wiring, scales to unlimited PMs, watcher lives on our infra. With company-managed backup accounts there is no personal-privacy issue. Caveat: this is the one option that is not strictly zero-backend — it adds a small server-side watcher + an OAuth connection — but it is reusable and off-device, and it is the most reliable path to scale.
3. Off-phone Mac watcher. iCloud Photos syncs the camera roll to a company Mac; a launchd folder-watcher there POSTs new captures. Watcher entirely off the PM's phone.

The capability triangle (pick two)

{ real-time · hands-free · no app } — any two, never all three:

• real-time + hands-free → native DAT app (preview-gated; and real-time only if the temple-button emits on the BLE publisher — still unverified, open question 6).
• real-time + no-app → per-photo voice command ("Hey Meta, send a photo…") — violates the no-voice-per-photo UX constraint.
• hands-free + no-app → cloud-backup webhook or Shortcut, accepting batch-on-dock. ← where the turnover flow actually lives.

How the iOS automation actually works (detail behind option 1)

• There is no native "photo added to library" automation trigger. The automation must hang off a proxy event (Bluetooth-connect, a manual one-tap at end of walk, time-of-day, or NFC tap), and the Shortcut figures out what's new.
• Shortcut logic: read last-sent timestamp from a file → Find Photos filtered on the Ray-Ban Meta EXIF camera tag and Date Taken after that timestamp → Repeat: either Send Message (MMS) or Get Contents of URL (POST, multipart, with an auth header + turnover id) → write back the newest timestamp (dedupe memory).
• HTTP POST beats MMS for us: silent (no per-send confirmation), carries the turnover id + full EXIF (GPS/timestamp), zero per-message cost. MMS may prompt per send (open question 2) and strips metadata.

Corrections to original assumptions

• "≤60-second self-serve setup" is optimistic. Personal Automations are not shareable via iCloud link — only the Shortcut is. The PM must manually create the automation (pick trigger → point at the Shortcut → disable "Ask Before Running"). That's the non-self-serve step and the main friction for non-technical PMs — a point in favor of the cloud-backup option, which has nothing to wire up on-device.
• No "your existing /api/glasses/upload endpoint." (Flagged because external assistants kept assuming one.) No such route exists. The real ingestion surfaces are the inbound Twilio MMS webhook and src/app/api/condition-reports/photos/route.ts (base64 → S3 evidence bucket → ConditionPhoto descriptor). Any HTTP-POST transport targets the existing endpoint, not a new one — consistent with this ADR's zero-new-endpoint stance.
• Don't fork storage to R2. Evidence photos must stay in the S3 Object-Lock bucket (the immutability invariant Slice 3 / deposit disputes depend on). A side-channel R2 bucket would bypass it.
• Meta cloud media: captures are also held in Meta's cloud for ~30 days, but there is no public API to pull your own glasses media from it — not a viable ingestion path.

Open questions — status after this pass

• (new, resolved) Can a Display Web App access the camera / photo library? → No. Native DAT only.
• Q1 (EXIF make/model strings): still open — ~5 min on real hardware; gates both the Shortcut filter and the cloud-backup watcher filter.
• Q2–Q5 (Shortcut send / automation / loop / latency): still open; partially mooted if the cloud-backup transport is chosen (no Shortcut, no on-device loop).
• Q6 (temple-button → BLE publisher emission): RESOLVED 2026-05-30 → NO. The temple button does not emit on photoDataPublisher; it writes to the native Meta gallery only. Real-time native capture still works — but via programmatic stream.capturePhoto() (an in-app button), not the hardware button. See Update — 2026-05-30.

Update — 2026-05-30: V2 native DAT proven end-to-end on real hardware

The V2 native DAT app was built and validated end-to-end on a real Ray-Ban Meta (Gen-2) — capture → upload → attach to a turnover, all confirmed server-side. Repo: PropFlow-Technologies/propflow-smart-glasses (native iOS Swift, XcodeGen, DAT 0.7.0). This resolves the load-bearing V2 question and corrects two assumptions in the original ADR.

Proven (on hardware)

• Real-time capture works. Wearables.configure() → startRegistration() → requestPermission(.camera) → createSession(AutoDeviceSelector) → addStream → stream.capturePhoto(.jpeg) → the photo arrives on photoDataPublisher immediately and POSTs straight to src/app/api/condition-reports/photos (base64 → S3 evidence bucket → ConditionPhoto), then …/turnovers/{id}/inspect. No folded/batch sync on this path — that constraint is a consumer-camera-roll behavior, not a DAT-stream one. Two real ~0.5 MB JPEGs landed 201 and attached to a turnover (200).
• No DAT preview admission required for development. The original ADR gated V2 on "DAT preview admission." It is not gated for dev/testing: Developer Mode (MetaAppID = 0 in the MWDAT Info.plist dict) registers on real glasses with app attestation skipped (callback returns attestationValidated=skipped&isDevMode=true). Admission / release-channels only matter for distribution (TestFlight, >self testers). Developer Mode is enabled in the Meta AI app: Settings → App Info → tap the version number 5×.

Q6 answered — the temple button does NOT feed the app

A user-pressed temple/capture photo does not emit on photoDataPublisher; it goes to the native Meta gallery / iOS Photos (momentsd activity, zero stream events on press). DAT 0.7.0 exposes no hardware-button, touchpad, or gesture API at all (verified across the full MWDATCore + MWDATCamera surface). The only capture trigger available to an app is programmatic capturePhoto(). So real-time + hands-free via the hardware button is impossible in 0.7.0; the capability triangle's "hands-free" leg now requires a built trigger (a phone-side voice command), not the temple button.

Video & audio — what DAT 0.7.0 actually exposes

• Video: yes. MWDATCamera.Stream.videoFramePublisher delivers frames; StreamConfiguration(videoCodec:resolution:frameRate:) is configurable (we ran .raw / .medium / 24). Viable for the senior-coaches-junior "see-what-I-see" use case. Exact max resolution/fps is still not SDK-pinned — the earlier "≤720p/30fps" stays a Meta claim, not confirmed. Per the 2026-05-28 mic-contention finding, video is blocked while on a call (it shares the mics).
• Audio: no DAT API. .microphone exists only as a permission enum value — there is no audio stream/publisher, no audio in StreamConfiguration, and no audio framework. The glasses' mic is reachable only as the 2026-05-28 pass already noted: a standard Bluetooth-headset input to the phone. So voice notes are a phone-side capability (iOS SFSpeechRecognizer on the glasses-as-BT-mic), not something the DAT app pulls.

Config gotchas that blocked it (for whoever picks this up)

Three of our misconfigurations — not the glasses, not attestation, not a paid account — produced a generic Meta "internal error / no callback" for hours:

1. Contradictory MetaAppID — MetaAppID = 0 at the plist top level and a non-zero production ID inside the MWDAT dict silently forces the production-attestation path a free sideload can't pass, because the SDK reads only the MWDAT dict. Fix: put all DAT keys exclusively in the MWDAT dict, with one consistent value.
2. Missing CFBundleTypeRole = Editor in CFBundleURLTypes — without it the <scheme>:// callback can't route back, so registration completes on Meta's side but never returns to the app.
3. Permission-flow hangs — requestPermission() throws requestTimeout and checkPermissionStatus() hangs until the glasses link is live. Detect the grant off the .onOpenURL callback (permission_granted=true) and timeout-guard every SDK call. Also: the glasses must be actively connected (out of the case, awake) in Meta AI before the camera prompt appears.

Net effect on the decision

V2 is no longer "post-MVP, admission-gated, may collapse." It is proven and buildable now for real-time capture — minus hands-free-via-hardware-button, which 0.7.0 cannot do. The turnover flow's remaining gaps are product work, not platform unknowns: (a) point the app at production + a real turnover + a device service-token; (b) a phone-side voice-notes layer (speech-to-text on the BT-mic) to replace the V1 voice-call narration; (c) an optional voice-command shutter for hands-free. V1 (Shortcut / cloud-backup) remains the zero-app path for PMs who won't install an app; V2 is the rich path for a managed fleet.

Alternatives considered

"Hey Meta, send <turnover line> a photo" per shot

Real-time send via the contact's WhatsApp / iMessage path. Rejected — per-photo voice violates the stated UX constraint.

DAT-native iOS app with BLE photo capture as V1

Deferred to V2. Requires DAT preview admission plus a TestFlight install on the PM's phone. Out of the 2-day budget; violates "no third-party app." Same backend contract as V1, so it swaps in cleanly later. Distribution reality (2026-05-28): no public App Store publishing during preview — install is via TestFlight / release channel, capped at ≤100 testers (per Meta's DAT FAQ), and TestFlight builds expire after 90 days, forcing re-install (Apple TestFlight). Fine for a pilot, painful as a real deployment until Meta opens open publishing. A minimal SDK-glue snippet looks small, but the real project scope — Xcode project, Apple Developer account, code signing, preview admission, and per-phone install — is far larger than any minimal code sample implies.

Display Web App as the photo capture / upload surface

A standalone in-lens Web App (HTML/CSS/JS, deploy-by-URL) that the PM invokes to send the last capture. Rejected — Display Web Apps have no camera or photo-library access (native-DAT-only; see Update — 2026-05-28). The Web App can render a HUD (checklist, status, "photo received ✓") but cannot reach a photo to upload, so it cannot be the transport. Held as a companion display surface for after Display hardware is in inventory, not a capture path.

Cloud-backup-account watcher (Google Photos / Dropbox / iCloud) → server-side webhook

Promoted, not rejected — see Update — 2026-05-28, "Better no-app transport options," option 2. The strongest no-app path for a real rollout: OS-level backup of the camera roll + a server-side watcher we own. Listed here for discoverability; it is the recommended V1.5 transport once past the 2-day MVP.

WhatsApp / Messenger video call with frame extraction on our backend

Continuous video stream via the Be-My-Eyes pattern; backend extracts frames on cue. Rejected — WhatsApp Business API does not accept inbound video calls; receiving them requires a headless WhatsApp account (Baileys-style), against WhatsApp ToS.

Facebook / Instagram Live → Live Video Graph API ingestion

Rejected — voice-triggered start, the stream lives in Meta's infrastructure, frame-extraction adds latency, and PMs would not livestream to a Meta-branded surface for property work.

Dedicated PropFlow work iPhone on tool belt

$60 used iPhone SE per property in kiosk mode, app pre-loaded; PM's personal phone uninvolved. Held as fallback if the Shortcut path proves unworkable in field testing.

Voice-only V1 with no photo channel

PM dictates findings with enough detail that no images are needed. Rejected — ADR-0034 Slice 3 mandates voice/photo-native capture from day 1; photo evidence is the most decisive piece in deposit disputes (Slice 4).

A new TurnoverFinding entity (initial draft of this ADR)

The first draft of this ADR proposed a new TurnoverFinding shape with source: 'glasses-voice' | 'glasses-photo'. Rejected during review — ADR-0034 already settles the capture data model. ConditionReport.capturedVia is already 'web' | 'voice' | 'sms' | 'system' — capture-source is metadata, not a type discriminant. Glasses produce voice and sms captures into the existing pipeline; no new entity, no new enum value.

References

Primary sources

• Meta Wearables DAT announcement (Connect 2025): https://developers.meta.com/blog/introducing-meta-wearables-device-access-toolkit/
• DAT public preview blog (Dec 2025): https://www.uploadvr.com/meta-wearables-device-access-toolkit-public-preview/
• DAT FAQ: https://developers.meta.com/wearables/faq/
• DAT iOS SDK: https://github.com/facebook/meta-wearables-dat-ios
• DAT Android SDK: https://github.com/facebook/meta-wearables-dat-android
• Meta — Build for display glasses (Web Apps + DAT public launch, 2026-05-14): https://developers.meta.com/blog/build-for-display-glasses/
• Meta AI app auto-import (folded-only confirmation): https://www.meta.com/help/ai-glasses/1510723473009423/
• Meta media storage / album behavior: https://www.meta.com/help/ai-glasses/1427588664906909/
• Ray-Ban Meta Gen 2 launch: https://about.fb.com/news/2025/09/ray-ban-meta-gen-2-better-battery-life-video-capture/
• Twilio US MMS pricing: https://www.twilio.com/en-us/messaging/pricing/us

Added in the 2026-05-28 research pass

• gHacks — Meta opens Display glasses to third-party developers (2026-05-18): https://www.ghacks.net/2026/05/18/meta-opens-ray-ban-display-glasses-to-third-party-developers-through-wearables-toolkit/
• Meta — cloud media on AI glasses (~30-day cloud retention; no public pull API): https://www.meta.com/help/ai-glasses/734190441863923/
• Ray-Ban Meta FAQ — mic in active use blocks capture; calls interrupt recordings: https://www.ray-ban.com/usa/c/frequently-asked-questions-ray-ban-meta-smart-glasses
• Meta — "share your view" on a video call (the live-stream exception): https://www.meta.com/help/ai-glasses/2144534879236727/
• Meta — sharing requires import to phone first: https://www.meta.com/help/ai-glasses/683425686669295/
• Dropbox webhooks + longpoll (cloud-backup watcher option): https://www.dropbox.com/developers/reference/webhooks
• Google Photos Library API (cloud-backup watcher option): https://developers.google.com/photos/library/guides/overview

Internal references

• Trello epic: https://trello.com/c/BoW61MAW
• Trello — Phase 0 hardware + experiments: https://trello.com/c/ZqkNO2N0
• Trello — V1 client-side capture UX: https://trello.com/c/W3GZepmz
• Trello — V1.5 DAT iOS app: https://trello.com/c/In1z0MTo
• Trello — V2 continuous video: https://trello.com/c/vqzTDWap
• Trello — Ecosystem reference card: https://trello.com/c/re0rEUtk