iOS app · Wearables

Meta Glasses Agent: my own assistant in Ray-Ban Meta glasses

Work in progress November 2026

Swift 6 iOS Wearables Claude ChatGPT
Type
iOS app plus a Swift package
Stack
Swift 6, Meta Wearables Device Access Toolkit
Tests
105 core, 4 UI
Status
Runs phone-only, glasses session not yet verified

The idea

I wanted to choose the assistant in my glasses. This project is an iOS rewrite of an open-source Android app (tupo_vision, MIT licensed): ask a question out loud, the app sends one frame from the glasses camera with it, and the answer is spoken back.

What it does

  • Voice in, voice out: on-phone speech recognition, a streamed model answer, and text-to-speech one sentence at a time.
  • One camera frame per question, taken from the glasses stream.
  • A temple tap pauses and resumes the conversation.
  • Bring your own model: sign in with ChatGPT, or use your own API key for Claude, kept in the Keychain.
  • Consent per provider, with stored proof and a Withdraw button.

How it is built

A conversation engine runs the loop through idle, listening, thinking, speaking and paused, and knows nothing about the glasses hardware. Model providers sit behind a protocol, so adding one does not touch the engine. The glasses connection uses Meta's Wearables Device Access Toolkit, and audio travels over Bluetooth.

Two rules are built in and not optional. The prompt tells the model never to identify a person from face, body or voice. And camera frames expire after three seconds, so an old view is never sent as the current one.

What I learned

  • Keep the main thread clean. Looking up a speech voice on the main thread froze the app. It now runs off the main thread with a time limit.
  • Not everything ports. The Android app's standby feature watches another app's media session, which iOS does not allow. App Intents are the planned replacement.
  • Hardware has prerequisites nobody mentions. Linking the glasses failed with an internal error until Developer Mode was on, and then failed again because the firmware was too old for the SDK.

This one is honest work in progress. It runs on a real iPhone in phone-only mode, and a full session on real glasses is the next milestone.

Want to know more?

This project is not public yet. Get in touch if you would like a walkthrough.